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Preface 


Debriefing is a major component of the job for personnel in many high-risk industries 
where errors can have tremendous, often deadly consequences, including combat, 
surgery, and aviation. Although there exists a great number of studies on debriefing, 
recent reviews of the research literature suggest (a) shortcomings in the topics 
researched, (b) paucity of related theory, (c) limitations in the number of empirical 
studies, and (d) problems in research design. There are also recent suggestions that few 
studies exist in the literature showing how to debrief, how to teach or learn to debrief, 
what the debriefed learn, what approaches to debriefing exist, and how effective these 
are with respect to their objectives and goals. Meta-analyses reveal substantial variations 
in research findings—e.g., on the use of video as a means of debriefing—that can be 
traced to methodical problems. 

Debriefing occurs in the context of the bi-annual recurrent training and assessment 
of pilots. Assessment, too, has been a thorny issue in aviation, as studies show 
considerable score variation even after raters have undergone substantive training (in 
one European study, there were three years of training involved). Whereas traditional 
approaches attribute these variations to error variance in statistical models, it is not 
at all clear whether score variation cannot be explained on rational grounds. Because 
human beings tend to be able to give good reasons for what they are doing and for 
their judgments, there may actually be good grounds for the observed variability of 
assessment scores between raters of the same performances. It is important for the 
airline industry to understand the rational source of these variations generally and that 
the variation may not be the result of measurement error but has its foundation in the 
different observations flight examiners make and in how they categorize these. 

This book redresses these problems in that it provides a detailed, cognitive 
anthropological investigation of debriefing and assessment, investigates the functions 
of different cognitive artifacts used, and provides theoretical frameworks that account 
for the complexity of flying an aircraft and for the debriefing of the pilots’ experiences, 
especially under the high-stakes condition of their bi-annual evaluation for licensing 
purposes. The book provides detailed investigations of flight examiners’ methods to 
arrive at assessments of aviation pilot performance. It shows, and theoretically models, 
why there are good reasons for the observed, lower-than-desired inter-rater agreements. 
The book offers detailed scenarios of how debriefing is and can be organized to draw 
maximum benefit for pilot learning, that is, for the take-home messages that make 
them better pilots. The theoretical framework includes objective factors that determine 
performance and the subjective experience pilots have while undergoing training and 
testing in flight simulators. 
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As the title suggests, this book is concerned with cultural practices and cognition 
in the field of aviation. There are two parts to this book, both including pilots and flight 
examiners as the main protagonists but involved in two distinct yet overlapping societal 
activities. In Part A, the focus is on flight examiners assessing pilot performance. To 
situate this work, expert-expert comparisons are made with other pilots at different parts 
on the career trajectory—captains and first officers—evaluating the performance of 
pilots. In this part readers find out about (a) flight examiners’ methods for arriving at an 
assessment, whether their airline does or does not use an explicit human factors model, 
(b) how pilots of all ranks reason in support of particular assessment scores that they 
assign to pilot performance, and (c) models that better explain the assessment process 
than the existing and predominantly used inter-rater reliability method. Part A, describing 
and theorizing pilot performance assessment, is concerned with the intersection of 
cultural practices and cultural forms of cognition rather than with the epistemological 
questions of (a) how to identify the mental representations assessors of pilot performance 
produce with respect to the primary phenomenon of pilot cognition or (b) how whatever 
happens in the debriefing room comes to be translated into practice—where it is hoped 
that the recurrent training and assessment have contributed to continuing professional 
learning and professional development. Inherently, cultural forms of practices and 
cognition are acquired by participating in relevant situations where they are made 
publicly available in and through the joint performances of participants. The work 
presented here therefore is aligned with a concrete human psychology that recognizes 
witnessable societal relations as the nature and origin of any specifically human form 
of behavior and higher psychological function (Roth 2016). 

Part B of this volume has debriefing as its focus. Again, flight examiners and 
pilots are the protagonists. But, in contrast to Part A, where the pilots examined are 
observed while doing their work, the second part has the two parties literally sitting at 
the same table. The content of their talk consists of what pilots have done, how they 
have done it, and what they may learn from their past experience. Debriefing meetings, 
quite apparently, are social phenomena. But saying that these are social phenomena 
because in each case there is more than one individual involved means taking the social 
in a weak sense. These meetings are recognizable to the insider not only because the 
contents of talk have to do with flying airplanes and the organization of this work but 
also because of the particular ways in which participants relate to each other. This is so 
because the meeting is the result of participants’ joint work, and each contribution is 
not only a form of expression but also is shaped to be understood by the mutual other 
and for the purpose of leading the other to act in specific ways. Each contribution to 
a debriefing meeting therefore is social in a strong sense because it takes into account 
how self and other work together to produce the debriefing meeting as a condition 
of their talk. That is, debriefing meetings are not ready boxes into which actors step; 
instead, they are endogenously produced, the result of the joint work of pilots and flight 
examiners then and there, and who relate to others so that the situation recognizably 
becomes a debriefing meeting. Each action, understood from a systemic approach to 
debriefing, is a social action, geared toward helping the respective other understand 
and toward making the respective other act in specific ways. Each word therefore is in 
common to all parties; if it were not, then the word would not have any transactional 
relevance—i.e., not affect others. Even if pilots and flight examiner did not know each 
other personally before, they would still pull off a meeting where debriefing recognizably 
occurs. The ultimate reason for this is because the practices and associated cognition 
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are cultural and historical in nature. Part B therefore provides descriptions of how 
debriefing meetings are organized in recognizable ways, how debriefing is conducted, 
how take-home messages are produced, and what role representational devices (tools) 
have in sense-making and learning. 

Each part of this book includes a longer introduction and contextualization to the 
chapters it contains. Throughout this book, reference is made to a number of aircraft 
incidences. Descriptions of these incidences are presented in grey boxes; in the 
preparation of these descriptions the original accident reports were used. The specific 
incidences include: (a) the collision with ground of a Fokker F100 operated by Air 
Bagan near Heho, Myanmar (December 25, 2012), in which a non-directional beacon 
(NDB) approach was bungled (Chapter 7); (b) a controlled flight into terrain of a de 
Havilland DHC-8 (Dash-8) aircraft operated by Ansett New Zealand (June 9, 1995) near 
Palmerston North, which followed the pilots’ confusion arising from a malfunctioning 
undercarriage and the failure to engage a go-around procedure after ground proximity 
warning (Chapter 8); and (c) the February 4, 2015 crash of an ATR-72-600 TransAsia 
Airways Flight GE235 in Taipei, Taiwan, parts of which had been captured on CCTV 
and dash-cams (Chapter 10). Details of some other crashes—including that of a 
military-operated B-737 near Dubrovnik, Croatia (April 3, 1996) and the ATR-72 212A 
of Lao Air Flight QV 301 near Pakse, Laos (October 16, 2013)—are also discussed 
in pertinent contexts. In addition, frequent reference is made to the case of a poorly 
executed missed approach (Chapter 1), which is the content of a videotaped scenarios 
employed in several earlier investigations (Roth and Mavin 2015; Roth et al. 2014a). 
Further boxes are associated with this scenario for the purpose of illustrating: (a) the 
idea of situation awareness! as a master concept (Chapter 2); (b) different approaches to 
the evaluation and assessment of pilot performance (Chapter 3); and (c) the fuzzy logic 
approach to modeling performance assessment (Chapter 4). Readers are encouraged to 
read those boxes even though they may be placed later in the book. 

Throughout this book, scenarios are used rather than direct quotations of individual 
pilots or flight examiners or descriptions of their actions. The scenarios are ethnographic 
devices to represent typical actions and responses (only Chapters 9 and 10 draw on 
actual, already published transcriptions though in a novel analytic context; they are 
referred to as “fragments”). In the approach employed here, focusing as it does on 
joint work and cultural practices, specific individuals and groups are considered to be 
staffing a social situation that is taken to be a phenomenon sui generis. Individual and 
groups of pilots and flight examiners do their work in recognizable fashion such that 
other pilots and flight examiners recognize themselves in this work and these actions. 
The work therefore never is merely particular but always the work of pilots and flight 
examiners in general. It is toward these general, culture-specific ways of acting and 
speaking that the present work is orienting, describing, and theorizing. 





' Throughout this book, the term situation awareness is used. In the field and some practically oriented 
academics, however, practitioners tended to use situational awareness or simply SA (e.g., Ford and 
Munro 2011; Mavin and Dall’ Alba 2010). 
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PART A 


CULTURAL PRACTICES 
AND COGNITION IN PILOT 
ASSESSMENT 


2 Cognition, Assessment and Debriefing in Aviation 


his first part of the book is concerned with the cultural practices and cognition 

of pilot performance assessment and includes, to situate assessment, a chapter 

reviewing different models of pilot cognition and pilot performance. This 
introduction begins with a brief look back at existing research on pilot performance 
assessment and a look ahead to the chapter contents that comprise this first part. A sketch 
of cognitive anthropology, the method that was used for all of the work presented here, 
concludes this introduction. 


LOOKING BACK 


In aviation, pilot training and assessment historically began with a focus on technical 
(flying) skills and associated technical aircraft knowledge and procedures. For obvious 
reasons, the assessment of the technical skills is an integral and essential part of 
performance assessment in aviation. The underlying assumption is that a focus on the 
technical skills in recurrent training and assessment will bring about increasing levels 
of flying competencies. Moreover, whereas pilots, as other practitioners, learn while 
doing their job, accidents tend to occur with considerably higher frequency when the 
aircrafts are in non-normal situations. That is, learning that occurs as part of the normal 
job does not prepare pilots for competently dealing with the abnormal situations. The 
advent of full-motion flight simulators made it possible to train flying in such situations 
that pilots rarely, if ever, encounter on the job. Thus, any recurrent training regime will 
include flying under adverse conditions, whether these adversities have their origin 
in the physical surrounding (e.g., weather, bird strike) or in the aircraft itself (e.g., 
engine fire at takeoff, undercarriage failure). This initial focus on the technical skills 
historically has led to a decrease in technical skills- and knowledge-related accidents 
in the airline industry. 

It turned out, however, that many accidents relate to actions that do not fit the 
classification into technical skills. In the 30-year period from 1959 to 1989, more 
than 70 percent of all aircraft accidents did not have technical failure as their main 
causes but instead were attributed to human actions (Merritt and Helmreich 2004). 
Technical proficiency alone, therefore, does not guarantee the safety of an aircraft. 
Thus, for example, communication and decision-making have been listed as problematic 
circumstances in airline disasters (Helmreich et al. 1999). Other cultural factors, 
such as the hierarchical relations between captains and first officers, which mitigate 
communication and especially the production of comments that are critical of a more 
senior pilot, also were involved in aircraft accidents. Thus, investigations revealed 
mismatches between traditional, technical skills-related training and assessment 
methods, on the one hand, and the possible non-technical skill-based causes of 
accidents, on the other hand. Such investigations therefore highlighted an important 
need for changes in pilot training and assessment (Salas et al. 2004). Historically, the 
notion of non-technical skill emerged for capturing a variety of types of actions that 
were different from and softer than the technical skills. The airline industry began to 
turn to crew resource management (CRM)' training, that is, to the training of pilots 
so they would better manage the resources that exist in the crew as a whole and the 
relation among pilots. This focus included, among others, the areas of decision-making, 





' Appendix A presents a list of all aviation-related acronyms used in this book. 
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situation awareness, management, and communication. Aviation organizations use the 
term crew resource management to refer “to the effective use of all available resources: 
human resources, hardware and information to achieve safe and efficient operation” 
and consider “CRM training [as] one way of addressing the challenge of optimizing the 
human/machine interface and accompanying interpersonal activities” (COSCAP-SEA 
2009). The topics to be included in CRM training within a three-year period include 
standard operating procedures, stress, and fatigue but also human factors concepts, such 
as situation awareness, workload management, decision-making, and communication. 

With respect to the non-technical skills and crew resource management, substantial 
global variations have been apparent in the implementation of instruction and assessment 
(Mavin et al. 2013). NOTECHS, a system for the classification of non-technical skills 
system, was developed in Europe for the purpose of assessing pilots’ crew resource 
management performances. The classificatory system includes cooperation, leadership 
and management, situation awareness, and decision-making as its major components 
(Flin et al. 2003). NOTECHS appears to be well integrated into airline practice. In 
countries like Australia, the focus on the assessment of non-technical skills NTS has 
been recent; and it was only halfa decade ago that the Civil Aviation Safety Authority of 
the Australian Government released its Civil Aviation Advisory Publication concerned 
with non-technical skills training and assessment for regular public transport operations 
(CASA 2011). The purpose of the advisory is to “provide advice and guidance for 
Australian Regular Public Transport operators to develop non-technical skills training 
and, where necessary, its assessment”. 

The NOTECHS framework divides the non-technical skill area into two main 
classes, social skills (cooperation, leadership and management) and cognitive skills 
(situation awareness, decision-making) resulting in four main categories (Flin et al. 
2003). Here the assumption is made that the social skills are directly observable whereas 
the cognitive skills have to be derived “since they do not directly materialise in overt 
behaviour but occur ‘in the head of the pilot””. Each of these categories is taken to 
be composed of four elements: team building and maintaining, considering others, 
supporting others, and solving conflicts. Finally, there are behavioral markers that 
manifest each element. For example, assessors might come to the conclusion that the 
first officer in the Missed Approach scenario (see box, Chapter 1) “supported the captain 
quite well; he was supporting the captain much more than the captain supported him”. 
They therefore come to the conclusion that the first officer should receive a positive 
check on the marker “interacted with crewmember and provided support”. 

The NOTECHS system has been subject to preliminary testing in an experimental 
setting (O’Connor et al. 2002). The study ends citing the positive opinion of the Joint 
Aviation Requirements Translation and Elaboration of Legislation (JARTEL) group. 
Accordingly, “the results of the experimental phase of the JARTEL project are quite 
encouraging for the further development and ultimate implementation of the NOTECHS 
method” (emphasis added). The authors also note that the instructors involved in the 
project were accepting and approving the approach. However, from a psychological 
perspective, the results were not so positive at all, for the inter-rater reliability and 
consistency measures were less than optimal. In other research (e.g., Mavin et al. 2013), 
too, there are considerable variations between the assessments different individuals make 
of the same pilot performances (e.g., as exhibited in brief, videotaped scenarios). As 
shown in the first part of this volume, however, there are actually quite good reasons 
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for the considerable variation in the assessment of pilot performance; such reasons are 
not taken into account in inter-rater reliability approaches. 

In part based on the perception that the NOTECHS approach overemphasizes 
the non-technical skills with respect to the technical skills, an approach was presented 
that is supposed to integrate both types into a single Model for Assessing Pilots’ 
Performance (Mavin and Dall’ Alba 2010). This model (MAPP) has two main classes 
of skills: essential skills (situational awareness [sic], aircraft flow within tolerances, 
decisions considerate of risk) and enabling skills (aviation knowledge, management, 
communication). The six categories encompassed by the two classes are arranged in 
three levels, with the three enabling skills at the base, and situation awareness at the 
apex, whereas the two remaining categories (tolerances, decision-making) constitute 
the intermediate level. Arrows generically point upwards from the base and downward 
from the apex. It therefore remains unclear what the nature of the relations between 
the categories is and why some of the categories should be considered skills rather 
than situated actions (e.g., is situation awareness a skill or does situation awareness 
describe a state?). Indeed, a subsequent study of the same database that has led to the 
MAPP—interviews with flight examiners—provides evidence that the so-called model 
is not a model in the psychological and human factors sense (Mavin and Roth 2014a). 
This second study shows instead that the six categories are part of flight examiners’ 
ways of talking about and doing assessment. At best, therefore, the originally reported 
“model” is a folk model. Indeed, the latter authors describe what they present to be a 
grounded theory, that is, the MAPP represents the inner organization of flight examiners’ 
talk—which is therefore nothing other than a folk model. Moreover, the second analysis 
shows that flight examiners actually connect the categories in more ways than the original 
publication leads to believe. Rather than situation awareness determining (1.e., somehow 
influencing) the two categories below it, flight examiners talk about how management 
and decision-making decreases the number of flight-relevant items that pilots are aware 
of within and outside the cockpit. That is, there should have been arrows pointing to 
situation awareness rather than merely arrows pointing away from it. 

The Model for Assessing Pilots’ Performance (MAPP) was used as the basis 
for developing a single-paged assessment form in which the six main categories 
are further broken down into a total of 20 subcategories (Mavin et al. 2013). Each 
subcategory is assessed on a scale from 1 (very poor) to 5 (very good). To facilitate 
the rating, the assessment instrument uses behavioral markers. For example, the main 
category aircraft maintained within tolerances has three subcategories: manual flight, 
automation, and monitoring. For the first of these, one of the behavioral markers reads, 
“aircraft manipulated to limit of tolerances, or slightly exceeded tolerance, immediately 
corrected”, which is found in the column under the score of 2 (poor). In the next column 
(3, satisfactory), the user finds the behavioral marker “manipulated with some deviation 
from target parameters, though quickly recovered”. The instrument was neither validated 
nor subject to experimental testing prior to being used in the field—where the research 
presented in Part A of this volume was conducted. Given the absence of any procedures 
that are normally used in the development of psychological instruments, it may come as 
little surprise to readers that there are considerable variations in the assessment and low 
inter-rater reliabilities even among flight examiners, who have received considerable 
training and who have had experience using the assessment tool. But, given that the 
NOTECHS system has the same problems, the issues leading to score variations likely 
are deeper. This is so because, as seen in Chapters 3 and 4, raters actually provide very 


Cultural Practices and Cognition in Pilot Assessment 5 


good reasons for these variations. It makes sense to assume that the assessment variation 
in the NOTECHS studies would have had good reasons had they been conducted using 
the same methods as the research on which Chapters 3 and 4 are based. 

The improvement of assessment requires better understanding of the assessment 
process. This is particularly the case if the goal is to contribute to an increase in crew 
resilience to surprises and anomalies in flight situations (Dekker and Lundstróm 2007). 
Part A of this volume is devoted to a better understanding of the variations not only in 
pilots’ performance assessment but also in the reasons provided for particular assessment 
scores. How do flight examiners assess pilots? What are the methods flight examiners 
use to arrive at their assessments? What are the reasons flight examiners use? What 
are the grounds on which particular assessment outcomes are based? Answers to these 
and related questions are provided in Chapters 2 to 4. 


LOOKING FORWARD 


How debriefing is conducted for the purposes of assessment and learning (professional 
development) depends on the (implicit or explicit) models used to understand the 
cognition and performance of pilots. Chapter 1 presents and discusses different models 
and ways of understanding used for theorizing what happens in the cockpit of an aircraft. 
Implicit in standard approaches to assessment and learning are se/f-action models, which 
decompose what happens on the flight deck into components so that errors may be 
attributed to individual pilots (e.g., Dekker and Hollnagel 2004). That is, the performance 
of the system is understood as the cumulative sum of individual performances, which 
includes take-up of previous performances. Current academic discussions problematize 
this approach, theorizing instead flight deck events in terms of interaction; the resulting 
models frequently are presented in terms of distributed cognition and joint cognitive 
systems. Critics of the traditional approach point out that what matters is the actual 
performance of a cockpit as a whole, whereas practitioners and traditional cognitive 
scientists insist on understanding the system in terms of individual skills. Both the self- 
actional and the interactional approaches discuss cognition in the abstract. Less in vogue 
are transactional models of cognition, where the nature and contributions of one part 
of a system cannot be understood independently of other parts of the system. That is, 
specifying the contribution of one part (e.g., pilot) requires accounting for other parts, 
which cannot be independently defined. Inherently, therefore, performance cannot be 
reduced to individual performance in transactional approaches. Such models, as these 
appear in the late work of the Russian social psychologist Lev Vygotsky, also include 
practical and affective dimensions. The category experience [perezivanie], which is 
a name for the irreducible person—environment unit, is an appropriate choice, for it 
combines objective external conditions and (subjective, inner) personal experience. 
Later, in aviation debriefing, instances of previous lived-experiencing (flight simulator) 
become the topic of a current lived-experiencing (debriefing). This category therefore 
allows theorizing not only what happens while pilots are flying aircraft but also what 
happens in debriefing (e.g., the joint work of flight examiners and pilots). 

When pairs of pilots are asked to assess the performance of a peer, very different 
patterns of approaches are observed between flight examiners, on the one hand, and their 
(with respect to assessment) less-experienced peers, on the other hand. The latter, if a 
scoring sheet is available, tend to use it to organize the assessment process and the actual 
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scoring procedure (Chapter 3). Flight examiners, on the other hand, use the documentary 
method of interpretation, a mundane form of reasoning, where observations are treated 
as evidence of underlying phenomena while presupposing these phenomena to categorize 
the observations. The description of the documentary method of interpretation constitutes 
the heart of Chapter 2. The chapter also shows how flight examiners take into account 
that cockpit events are the result of the work of a joint cognitive system and what they 
do to tease out the performance of one of its component (i.e., one of the pilots). The 
do so by implementing what psychologists have recommended for assessing aptitude: 
follow a person across many different performance situations. 

Historically, those interested in the assessment of practitioners generally and in 
the assessment of aviation pilots specifically focus on the assessment outcomes when 
formal instruments are used. Little is known, however, how assessors arrive at their 
conclusions and what assessors take into account prior to marking a particular event. 
Moreover, little is known about how pilots at different stages in their career assess their 
peers. Chapter 3 is concerned with answers to questions including, “What are pilots’ 
reasons for assessing the performance of their peers?” and “Do the reasons and ways 
of assessing change with the assessors’ amount of experience flying aircraft?” That is, 
this chapter presents an ethnographic study of reasoning as it is used in the course of 
assessment of pilot performance. A description is provided of what has been learned 
about the assessment process and assessment outcomes when pilots at different stages 
of their career—first officers, captains, and flight examiners—working in same-status 
pairs, assess videotaped scenarios in two conditions: with or without an explicit human 
factors model. 

Existing research shows that the assessment of practical performance is difficult, 
especially when the focus is on non-technical skills. Researchers and practitioners find 
that when measuring instruments are used, inter-rater reliability tends to be moderate 
to low. In this approach, the variations in assessment scores are treated as error variance 
arising from inconsistencies in the ways in which assessors (flight examiners) evaluate 
situations. However, Chapter 3 shows that pilots at all levels provide each other with 
intelligible, plausible, and convincing reasons for assessing pilots in one rather than 
another way. Even when they discuss their observations to arrive at a shared assessment, 
the scores that different evaluator pairs attribute differ substantially and considerable 
variation can be found in the reasoning about whether a pilot should pass or fail. Chapter 
4 explains the phenomenon, suggesting that rather than being a measurement issue, 
assessment is done as categorization on the basis of the factual evidence collected and 
taken into account. Three different approaches are described: constraint satisfaction 
network, catastrophe theory, and fuzzy logic. Whereas the first two types of models 
are presented particularly for their affordances as metaphors to understand how minor 
quantitative variations can lead to qualitatively different assessment outcomes, the 
fuzzy logic approach is worked out in greater detail. The fuzzy logic model predicts 
very well which scores assessor pairs actually give. The approach also is conceptually 
easy and has few computational requirements so that it can be done using a spreadsheet. 
The chapter ultimately shows that the inter-rater variations in assessment are not due 
to error but can be explained. 
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COGNITIVE ANTHROPOLOGY 


Classical models of cognition theorize knowing-how and knowing-that in terms of 
mental representations and procedural knowledge stored in long-term memory. When 
required, these representations are called up, brought into short-term memory and 
applied to the situation at hand. When people talk about situations, they are said to 
externalize their cognitive models, representations, and cognitive structure by means of 
words, gestures, and, where applicable, other, often written communicative forms (e.g., 
whiteboard drawings and text). However, in talking, pilots and flight examiners do not 
just spill the contents of their minds. They talk (a) in response to what another person 
has said, (b) for the purpose to be understood, and (c) in turn, in-order-to get the other 
act in particular ways. That is, language and the organization of debriefing meetings are 
cultural practices produced to be intelligible to others and thus are social through and 
through. There now exists a vast literature on the dependence of cognition not only on 
culture but also on the very (societal) activity in which it occurs. In fact, sociocultural 
theories presuppose that every higher cognitive (psychological) function was a social 
relation first and, therefore, is a societal phenomenon that has cultural and historical 
dimensions (e.g., Leont’ev 1978). To properly understand cognition and cognitive 
development, culture needs to be taken into account. Indeed, reasoning is a social 
phenomenon through and through. Even when a people are on their own, what they do 
and how they do it is social, including writing in and reading personal notes (Vygotsky 
1989) or crossing lonely four-way intersections (Livingston 2008). The study of culture 
is the domain of anthropology. The interdisciplinary approach investigating cognition 
and culture simultaneously is referred to as cognitive anthropology (D’ Andrade 1995). 

Much of the existing psychological research makes use of experimental, 
correlational, and survey methods. The search for overall trends, however, tends to 
make invisible the mitigating effect of other psychological aspects and even more so 
those of cultural nature. The research on which the present volume is based made use of 
methods as appropriate, including not only those generally referred to as quantitative and 
qualitative but also mathematical modeling. This is especially important given that the 
studies represented here investigate “cognition in the wild” (Hutchins 1995a), describing 
and theorizing how flight examiners assess pilot performance and how flight examiners 
and pilots actually debrief in the course of their everyday work. That is, the methods 
were chosen to be appropriate to the phenomena. It cannot be otherwise, for “the most 
‘empirical’ technical choices cannot be disentangled from the most ‘theoretical’ choices 
in the construction of the object” (Bourdieu 1992). Rather than doing research so that 
it fits the Procrustean bed of the one or two methods known to or preferred by specific 
investigators, thereby skewing the phenomena, the present work employs methods from 
the entire spectrum of frequently polar “qualitative” and “quantitative” methods; and it 
sometimes combines them to achieve a mixed-method approach. This way takes into 
account the generally qualitative nature of cultural practices and how these operate in 
cognition; and it takes into account the reverse effect of cognition on culture. Whereas 
experimental methods are well suited to laboratory settings, researching debriefing 
outside the laboratory requires appropriate methods—if the results are to make any 
sense. Cognitive anthropology, which combines the methods of traditional psychology 
with the toolkit available to anthropologists, affords investigating phenomena otherwise 
excluded to the research community. 
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The research methods used here included, among others, modified think-aloud 
protocols to elicit the reasons pilots of all ranks use when asked to assess the performance 
of pilots shown in brief videotaped scenarios. Here, the use of the same scenarios that 
all assessors rated controls the object assessed and, therefore, allows for comparisons 
that are virtually impossible when observing assessment “in the wild”, where no 
situation observed in a flight simulator occurs twice. Normally, think-aloud protocols 
are conducted using individuals (experts or novices). But this method forces research 
participants into doing what they do not normally do: talking about their thinking while 
they are thinking. Indeed, psychologists have known for quite some time that the very 
fact of speaking develops thought rather than representing it (Vygotsky 1987). On the 
other hand, when asked to assess pilots while working in pairs under the condition that 
the pairs have to arrive at a single assessment, the participants inherently begin to talk 
to each other, exhibit their reasons for opting for one rather than another assessment, 
state the bases of their reasons, and so on. This modified method therefore constitutes a 
naturalized approach to the traditional think-aloud protocol that participants often find 
awkward and unnatural—thus skewing and biasing the data produced. 

Another example of the principled choices of method underlying the present work 
is the way in which the debriefing research was designed. The original hypotheses 
included anticipated differences that should be observable as a function of the existence 
or not of an explicit human factors model that airlines use. This factor was crossed with 
another one that took into account whether participating airlines make or do not make 
use of a debriefing tool. This therefore created a 2 x 2 factorial design that afforded 
quantifications and statistical inferences to be made (e.g., Roth 2015a, 2015b). Other 
mathematical approaches are used in Chapter 4 for the purpose of mathematically 
modeling the assessment of pilots. 

To understand debriefing processes and teaching | learning that occur in debriefing, 
pre-post measures and summary analyses that are inherent in statistical approaches are 
less useful. Here, methods typical to anthropology and applied linguistics are much 
better suited to produce scientific knowledge. The purpose of such analyses is to develop 
ethnographically adequate accounts (McDermott et al. 1978) of debriefing or teaching 
| learning in the context of the (continued) training of airline pilots. In this approach, 
we, researchers, “use the ways members have of making clear to each other and to 
themselves what is going on to locate to our own satisfaction an account of what it is 
that they are doing with each other” (original emphasis). One of the representational 
technologies for presenting such ethnographic research exists in vignettes. These tend 
to be descriptions of what people do, presenting the essence of a cultural practice, and 
have their origin in multiple actually observed cases. That is, vignettes are composites 
that allow members to recognize typical behaviors of their setting all the while protecting 
individual research participants from being identified (Bradbury-Jones et al. 2012). 
Throughout this book vignettes are use in this manner for the purpose of depicting the 
cultural practices that can be observed in the aviation industry. 
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e How the assessment of pilots is conducted—and ultimately debriefing for the 
purpose of learning (professional development)—depends on the (implicit or 
explicit) models used to understand cognition and performance. All cognitive 
models can be categorized according to the unit of analysis that is taken for 
understanding cognitive events. 


e In self-actional models, a pilot is characterized by procedural and declarative 
knowledge that is encoded in the mind (brain). In specific situations, this knowledge 
is applied to the problem at hand. Barring any breakdowns, what pilots do should 
be predictable beforehand based on the declarative and procedural knowledge. 
Some possible failures in the application include that of transfer, where a skill 
exhibited in one situation is not transferred to and applied in a second situation. 
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e Researchers using interactional models recognize that the performance of an 
aircraft as a whole cannot be understood based on the cognitive structure of the 
individual pilots and the structures of the cockpit and the underlying computing 
technology. Instead, the cockpit as a whole is understood to constitute a single 
cognitive system. Information is passed between the different components, some of 
which are human (pilots) whereas others are technological in nature. Interactional 
models are often described in terms of self-actional elements of the system, which, 
when combined by means of the information that is passed back and forth between 
the elements, constitute the system as a whole. The system then is understood as 
a joint cognitive system and in terms of its distributed cognition. 


e Transactional models acknowledge that the system at hand cannot be described in 
terms of self-actional elements. Each part of the system is part of the description 
of other parts. It is therefore impossible to understand any individual part on its 
own, that is, alone or in its relation with one or more other parts. To understand the 
person characteristic of pilot performance requires understanding the characteristics 
of the environment; and to understand the role of the environment (other pilot, 
technology) characteristics on pilot performance requires understanding the person 
characteristics. 


e Experience is a category that combines pilot and environment into a single, 
irreducible unit. This category will be important in understanding debriefing, an 
event in which there are opportunities for making past lived-experiencing the 
object of present lived-experiencing. It is precisely through the lived-experiencing 
of prior lived-experiencing that learning and development occur. 


e Activity is a category that contains all the characteristics of society. It includes 
all aspects of a workplace situation, such as the tools of production, division of 
labor, and the operating rules. Normally used for productive processes—e.g., 
manufacturing cars, producing grain, or hospital services—it is equally useful 
to understand what happens in the cockpit or the events in the debriefing room. 
Activity is understood as culturally and historically situated. 


e Folk models are the commonsense ways of understanding characteristic of 
practitioners. Folk models distinguish themselves from scientific models by their 
descriptive nature and the absence of rigorous forms of explanation. Much of 
performance assessment in aviation is based on folk models even when they are 
formalized into graphical representations and associated assessment instruments. 


his book essentially is about cognition and practice in aviation. The term cognition 

denotes what and how people know and learn; and practice refers to what people 

do and how they do it in recognizable ways. The present volume focuses in 
particular on the work of flight examiners, who evaluate and debrief pilots as part of 
biannual recurrent training and assessment cycles. Cognition and practice, therefore, 
appear at two levels. On the one hand, our interest lies in how flight examiners do what 
they do and the cultural practices that they observably participate in and observably 
display in what they do. But in evaluating pilot performance and in assessing pilots, 
flight examiners not only determine how well pilots do their job particularly in the flight 
simulator-based examination sessions but also make judgments about pilots’ levels of 
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knowledge and skills. In so doing, flight examiner produce explanations that make use 
of theoretical discourses that include concepts developed in the field of human factors, 
ergonomics, and psychology—though they may do so in ways that are more typical 
of laypersons. It turns out that the flight examiners’ own cognition and practice is a 
referent for judging and assessing the flying performances of others. 

How cognition is understood and modeled depends on the categories used, the 
smallest units of analysis into which human behavior can be decomposed while 
preserving the characteristics of the relevant whole. In recent years, there has been a 
shift from units defined by properties inherent to the individual brain to units defined 
by relations and dynamic patterns that include the brain, body, and environment 
leading to the study of cognitive ecosystems. In fields of inquiry denoted by such 
labels as “situated cognition” or “distributed cognition”, the question is not whether 
things and relations outside of the brain think and whether thinking occurs outside 
the brain and is distributed across agents and environments (Roth and Jornet 2013). 
The assumption that the brain 1s involved is never questioned. Instead, the question is 
whether the exclusive look into the brain case is sufficient for understanding complex 
behaviors, such as those that get an aircraft from point A to point B. In choosing one 
unit of analysis over another, the investigator actively defines the boundaries of the 
phenomenon and, thereby, situates cognition and in this manner frames the domain that 
is theorized (Roth 2001b). Categories of analysis are determined by the framing. Thus, 
if researchers choose to define cognition exclusively in terms of processes that might 
occur in the brain case, and by making some other assumptions, they end up with the 
classical viewpoint of cognition that uses a digital computer metaphor. Alternatively, 
using some other presuppositions, researchers might end up with a viewpoint that uses an 
artificial neural network and parallel distributed processing metaphor. The understanding 
of how an organism learns is tied to the chosen theoretical model; and these models 
are essentially different already when choosing computing metaphors, because digital 
computers and artificial neural networks learn and know in radically different ways. In 
good old-fashioned artificial intelligence, computers do not learn; they are programmed. 
Artificial neural networks and specifically those implemented in autonomous robots, 
learn by interacting with their environment, which also provides them with feedback. 

Issues become even more complex when researchers realize that the environment 
is the origin of any particular shape of behavior or imputed knowledge. Thus, for 
example, the particular ways in which the hands typing this text move are only in part 
due to the structure of the human hand. The QWERTY keyboard with its particular 
layout and dimensions has shaped the coordinated movements of the two hands with 
their ten fingers. The hands move without requiring conscious awareness to find the 
appropriate keys to be pushed down. In fact, they type in the manner a person speaks, 
writing words before a sentence has taken shape in the conscious mind of the writer. 
Even more curious, hands have been observed typing in English for short periods 
while the person has a conversation in another language. To understand such capacities 
appropriately requires units that are larger than the brain case; and these units need to 
be studied historically and culturally. Whatever happens in the brain at the same time, 
therefore, cannot be understood in terms of mental processes alone, for the QWERTY 
keyboard and the historical way in which it shaped the human body typing these lines 
are outside of the brain. But what happens in the brain is such that it supports typing 
on the QWERTY keyboard. This is why some neuropsychologists are taking a cultural- 
historical approach, recognizing that human activity not only is driven by individual 
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goals but also has societal, historical, and cultural dimensions, on the one hand, and 
has associated brain processes, on the other hand (e.g., Luria 1976). Historically, 
those brain processes responsible for typing words and sentences have been shaped in 
relations with the keyboard; they have their origin in experiences with the keyboard. 
Now, in typing this text, these same brain processes are triggered again when the writer 
is confronted with the task of writing while sitting next to and operating the keyboard. 
In holistic approaches, the activity of individual or group of neurons firing while the 
letters appear on the monitor is understood as an integral and constitutive part of a 
system, which includes the keyboard outside of the body, rather than as the result of 
the brains own internal processes. 

The chosen unit of analysis determines what is to be understood as cognition and 
what lies outside of it. Some researchers might talk (write) about the brain processes 
involved in typing independently of the task and tool. Others recognize the historical 
shaping of the processes and the situated triggering of the processes; both shaping and 
triggering processes are outside the brain case. Those describing their work in terms of 
situated or distributed cognition would analyze typing as a process distributed across 
the typist and the keyboard. Different models bear different implications for how the 
notion of representation is understood. The notion of distributed cognition, for example, 
has arisen as the result of taking as the unit of cognitive analysis ship navigation, 
a task involving a team of people and the coordination of several representational 
devices (Hutchins 1995a). The result is a joint cognitive system where the workings 
of individual parts can be understood only in their relation to the functioning of other 
parts. Different approaches to the study of cognition can be drawn in regard to how 
integral the notions of body and environment are to the minimal unit of analysis. The 
various existing models of cognition thus can be classified according to the following 
provisional characterizations: 


e Self-action: where things are viewed as acting under their own powers. 

e Inter-action: where thing is balanced against thing in causal interconnection. 

e Trans-action: where systems of description and naming are employed to deal 
with aspects and phases of action, without final attribution to “elements” or other 
presumptively detachable or independent “entities”, “essences”, or “realities”, and 
without isolation of presumptively detachable “relations” from such detachable 
“elements” (Dewey and Bentley 1949/1999). 


The first two characterizations describe the predominant number of theories 
currently in use, including those that attempt to integrate body and mind into a single, 
irreducible unit—often referred to by such names as embodiment and enactivist 
theories—and those that attempt to account for the social nature of both scientific 
and vernacular (folk) knowledge. Transactional approaches, on the other hand, are 
very rare indeed, which may be because they require going beyond the individual and 
its cognition, understanding these as effects of more extended processes and events. 
Such approaches cast the net for understanding cognition wide enough to capture the 
organism-acting-in-the-environment-as-a-whole, a system that is continuously unfolding 
in historical and cultural terms. Such approaches recognize that any psychological 
function, any specifically human action or characteristic, is the ensemble of societal 
relations (Vygotsky 1989). That is, the human essence practically exists in the ensemble 
of relations that constitute society. 





Models of Pilot Cognition and Performance 13 


Missed Approach 


The Missed Approach scenario described here was used in several research projects, 
where pairs of pilots at different levels in their careers—first officers, captains, and 
flight examiners—rated the performances of the two pilots (Mavin et al. 2013; 
Roth and Mavin 2015; Weber et al. 2016). There were two conditions. The larger 
study included pilots from airlines that (a) used an assessment model and associated 
assessment tool and (b) did not use an explicit assessment model or assessment tool 
but simply rated the performances as pass or fail. In all instances, pairs of same- 
ranked pilots were asked to provide reasons for their assessments (see Chapter 3 
for details on the studies). The scenario was recorded on the respective aircraft type 
that the pilots were routinely flying. 

In the Missed Approach scenario, the captain is the pilot flying (PF) and the first 
officer functions as the pilot monitoring (PM). The aircraft approaches an airport 
in poor weather, which includes low cloud and rain near the airport, though there 
also are openings in the cover. The pilots fly an instrument landing system (ILS) 
approach. Under the prevailing conditions, a circle-to-land approach is conducted 
even though it may be the most hazardous of approaches (as per the experts evaluating 
the pilots as part of the research conducted). The research among the airlines shows 
that pilots experience difficulties with flying the missed approach following the 
circling maneuver. Although the standard operating procedures call for an initial 
turn towards the runway before establishing the missed approach track, training 
managers report that a considerable number of pilots do not distinguish between 
the information on the approach chart and the flight path that they ought to follow 
from the point onward where the missed approach is called (e.g., wanting to turn 
right, as per chart, when in fact they have started the circling part, at which point 
the turn has to be to the left). 

The circling approach in the scenario has the aircraft descending safely on a 
predetermined route, using cockpit instruments to a specified altitude called minimum 
descent altitude (MDA). As the map of the approximate scenario flight path shows 
(Figure 1.1), the aircraft initially comes straight toward the runway and just after the 
runway is visual, turns right and eventually flies parallel to the runway for a while. 
The pilots prepare the aircraft for a landing on the chosen runway. However, just 
when they begin turning the aircraft to circle, they no longer have visual reference to 
the runway. The aircraft enters a rain shower, an instrument meteorological condition 
(IMC). The pilot flying appears to ask, “Missed approach was right hand, wasn’t 
it?” (see Scenario 1.4 below). The PM corrects by forcefully suggesting that the turn 
to be conducted is to the left. The PF replies by saying something like, “So it was 
left?” prior to confirming and executing a left turn. As the map shows (Figure 1.1), 
there is a mountain range to the right of the aircraft, which would have increased 
the risk of crashing. 

The research reported in Chapter 3 shows that experts rating the scenario pilots 
differed with respect to their perceptions as to what has happened. Certain facts about 
the flight, however, are clearly established and commonly agreed upon: 


+ The PF never brings the aircraft onto the required path of descend and instead 
remains consistently too high; and the aircraft also tends to remain too fast during 
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Figure 1.1 Flight path of the aircraft in the Missed Approach scenario. The aircraft encounters an 
instrument meteorological condition just after the final turn onto the runway has begun. The dotted 
path corresponds to the part of the flight following the announcement of the go around. 


the initial part of the approach. As a result, the aircraft is closer to the airport 
than required by the standard operation procedure when it reaches the minimum 
descent altitude (MDA) of 1,100 ft. 

After having turned the aircraft onto the circling pattern and prior to flying parallel 
to the airport, the PF notes that there is bad weather in the area where the final 
turn of the approach is to be made; the PF says that they have to watch out. The 
crew does not deliberate at this point the possibility of losing visual reference 
and does not discuss alternatives in such an event. 

After the aircraft has begun the final turn, both pilots appear to be caught by 
surprise when they find themselves in an IMC condition. 

The gyroscope shows that the aircraft, which was banking to the left due to the 
circling maneuver, returns to a bank angle slightly to the right and towards the 
mountain. 

The specified go-around procedure, which would take the plane back up to 
acceleration altitude and out of danger, is not flown according to the one stated 
in the airlines” standard operation procedures. As an example of such a procedure 
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shows (Table 1.1), there is an ordered sequence of events distributed across the 
two pilots. Apart from the question (confusion) about whether to turn right and 
the correction to fly left, the PF also does not press the go-around button (located 
on the power levers, as seen in Figure 9.2), as called for in the procedure. 


The scenario ends with the aircraft being made to climb to the missed-approach 
altitude. While the aircraft is climbing, the PF (captain) articulates being sorry about 
thinking that the turn was to be to right, and the PM also acknowledges to have been 
confused. He does so even though he has been quite assertive in the way in which 
he announced that the turn needed to be to the left. 

This scenario shares an important fact with the deadly incident of Lao Air QV 
301, which left dead all 49 people on board (LPDR 2014). The report cites as probable 
cause of the accident “the sudden change of weather condition and the flight crew’s 
failure to properly execute the published instrument approach”. The aircraft was 
mostly in the clouds during the final portion of the flight. The pilots conducted a right 
turn upon missed approach instead of following the (nominal) climb published in 
the VOR/DME approach chart of Pakse (Laos) airport. The flight director crossbars 
were consistent with the operating logic of the aircraft systems, but inappropriate 
for the go-around. Although the term situation awareness is not used (the summary 
report focuses on management issues), several associated perceptual factors are 
included: somatogravic illusion on the part of the PF, the PM’s tunneled attention, 
and the aircraft descended because the pilot followed the flight director bars rather 
than climbing (as the scenario pilot does in the same configuration). 


Table 1.1 Excerpt from a Typical Missed Approach and Go-around Procedure. 








Flight Event PM PF 





ANNOUNCE: 
“Go-around, set power, flap 15” 


Runway not in sight 





DO: DO: 
e Flaps 15 SELECT (A) * Go-around button press (A) 
* Torques CHECK (B) » Power levers ADVANCE (B) 
e Rotate pitch to 8° NOSE UP (C) 
* Cavalry charge CANCEL 
ANNOUNCE: 
“Flaps 15, power set” 
Positive rate ANNOUNCE 
“Positive rate” ORDER 





“Gear up, heading, low bank, IAS 


DO: » 
» Gear level UP (C) > 
e Hdg Mode SEL (D) 
+ Lo Bank SEL (D) 
e IAS VGA SEL (D) 
* Taxis & T/O Light OFF (E) 
ANNOUNCE ANNOUNCE: 
“TAS xxx set” “Check” 
ORDER & DO: 
DO & ANNOUNCE: “Set speed bug VGA” 


e Set speed bug VGA 
“Set” 





» Set speed bug VGA 
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SELF-ACTIONAL MODELS OF PILOT PERFORMANCE 


Research in aviation generally takes the traditional approach to cognition, which is 
resumed to the information processing that occurs in the mind of the individual operator 
(Martins 2016). Because the elemental units of this model (representations) are used to 
explain the behavior of what is defined as the cognitive system, this model is referred to 
as self-actional. In this view, each pilot is a complete system that tends to be modeled 
using the metaphor of the digital computer. All actions of a pilot then are the result of 
computations that occur in the mind and on the basis of, and producing, an internal 
mental model (Endsley 1994b). In such a model, situation awareness sits at the boundary 
between the mind and the environment (Figure 1.2). Some scholars do in fact consider 
situation awareness to be “adaptive, externally directed consciousness” (Smith and 
Hancok 1994, original emphasis). Viewed in this way, the mind is a (possibly shaded) 
mirror of what happens outside in the world. Based on this internal model, the mind 
directs attention to specific parts of the environment based on the current mental model; 
and “situation awareness [is] an operator’s internal model of the surrounding world” 
(Endsley 1994a). Actions, too, are situated at the interface between the pilot and the 
surrounding world because (a) they are the result of an enactment of existing mental 
scripts or are invented on the spot and at the same time (b) change the environment. 
The situation model affects the overall mental model, whereas the overall mental model 
simultaneously directs comprehension, projection, and expectations of the situation 
model. 

In the classical approach to cognition, the (sensory) experience in one or more 
situations gives rise to representations on which the mental model is based and 
the ensemble of which constitute the mental model (Roth and Jornet 2013). The 
representations are the results of abstractions from the concrete situation, which are 
derived from the situation model produced in situation awareness (Figure 1.2). These 
representations, because they are abstractions and therefore are not tied to the specific 
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Figure 1.2 This cognitive model features situation awareness as a central part, and it is therefore suited 
in the case of aviation practice where situation awareness functions as a master concept (Chapter 2). 
The arrows indicate the directions of (information, action) flow. 
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situation, can be used in (applied to) all other situations that are consistent with the 
deep structure retained in the abstraction. Representations are symbols stored in long- 
term memory in the form of declarative knowledge and procedural knowledge. It is the 
latter—existing in part in the form of scripts—that determines the specific actions to be 
executed to transform the environment. The more abstract the representations stored in 
long-term memory, the larger the set of mental models in which these representations 
can be used. From this view, relations with the environment involve the enactment of 
production rules—abstractions of functional relations with the environment—that are 
activated by environmental stimuli (see box). In “turning to the left” while driving a car, 
for example, cognition involves the enactment of production rules of the type “IF the road 
curves to the left THEN turn to the left”. The body is considered, as the environment, 
to be the “raw sensory input” for cognitive processing. Cognitive phenomena include 
encoding, retrieval, or processing of information. 





Procedural Knowledge 


In self-actional models, performance is theorized in terms of the knowledge and 
skills of an individual. This knowledge is thought to exist in long-term memory, 
from where it can be called up, entered into working memory, and used for the 
purpose of the situation at hand. Two forms of knowledge tend to be discussed: 
(a) declarative knowledge (episodic, semantic) and (b) procedural knowledge. In 
self-actional models, both of these forms of knowledge are contained in long-term 
memory; they are central parts of what may be called the cognitive layer (Figure 
1.3). Declarative knowledge tends to be treated in aviation theory and practice 
under “technical skills”, though in the case of the standard operating procedures of 
an airline, it would be insufficient for acceptable performance to merely remember 
the procedures as sets of facts. Other “non-technical skills”, including “decision 
making”, “managerial skills”, “cooperation”, or “communication” clearly fall into 
what we might call what-do-you-do-if domain. In the cognitive sciences, problem 
solving has been modeled using production systems (Anderson 1985). 

In the simplest case, a production consists of a goal and an action; in more 
elaborate cases, one or more tests of application will modify the production. 
Procedural knowledge, knowledge of how to do things, is modeled in traditional 


cognition working interface environment 
buffer 





vision rain cloud 


E press GA button, rotate to 8°, 
declarative y PL to ramp, cancel CCH 
~SOPs action 


hearing 


procedural 
- Go-around “Go-around, set power, 
- Positive rate | checking „flaps 15” 
of climb 


speaking 


Figure 1.3 A production system models the procedural knowledge of the pilot flying during a 
missed-approach procedure, here the first part of it referred to as “go-around”. 
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cognitive science by sets or systems of productions that accomplish certain task. 
In the case of some problem, rules are called up from long-term memory and, one 
after the other, are entered in to short-term memory where they can be actioned 
on. That is, when the pilot flying the Missed Approach scenario recognizes that 
what is happening outside the cockpit window—e.g., an instant of an instrument 
meteorological condition that no longer allows visual contact with the runway or 
some other unexpected event—he is supposed to abandon the landing approach 
and fly a go-around procedure (Table 1.1). In aviation practice, all pilots have a 
copy of the manuals that describe all standard operating procedures (SOPs) for the 
operation of the aircraft they are flying. (The manufacturer also has a flight crew 
training manual in which all normal procedures are listed, including the one for a 
missed approach.) The beginning of the procedure in the case of the missed approach, 
the first set of productions pertains to the “go-around”. To implement the first steps 
of the standard operating procedures (Table 1.1), these productions (Px) might be 
modeled in the following way. 


Productions for “Go-around” 
Goal Setting 


P1 IF Instrument meteorological conditions or other unexpected events 
THEN set goal fly sequence “Missed approach” 


P2 IF the goal is to fly sequence “Missed approach” 
THEN set subgoal 
1. Fly sequence “Go-around” 
2. Fly sequence “Positive rate of climb” 


P3 IF the goal is to fly sequence “Go-around” 
THEN set subgoals 
1. To make announcement “Go-around, set power, flaps 15” 
2. To do depress go-around button 
3. To rotate aircraft to 8° nose up 
4. To move power levers to ramp 
5. To cancel cavalry charge 


Action Execution 


P4 IF goal is to make announcement “Go-around, set power, flaps 15” 
THEN ANNOUNCE “Go-around, set power, flaps 15” 
P5 IF goal is to do depress go-around button 
THEN DO depress go-around button 
P6 IF goal is to rotate aircraft to 8° nose up 
THEN DO rotate aircraft to 8° nose up 
P7 IF goal is to move power levers to ramp 
THEN DO move power levers to ramp 
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P8 IF goal is to cancel cavalry charge 
THEN DO cancel cavalry charge 


Figure 1.3 conceptualizes the structure in operation during an event initiated, 
for example, when the aircraft enters a rain cloud upon circling. In the participating 
airlines, distinctions were made between “what happens outside [the cockpit” and 
“what happens inside the cockpit”. As long as what happens inside a cockpit comes 
from a source outside of the skin and brain case of the pilot, it was taken as outside 
information—thus part of what Figure 1.3 conceptualizes as “environment”. Ifa pilot 
“misses” something, such as the fact that there is a mountain (range) to the right of 
the final semi-circle (Figure 1.1), then the flight examiners tended to attribute the 
failure to the “non-technical skill” “situation awareness”. 

The purpose of the annual proficiency check is to ascertain that whatever is on 
the inside of the cognition layer is sufficient for the safety of an airline’s operation. 
However, flight examiners do not have access to this layer. Moreover, they have 
neither the time nor the technology to look into the brain case; they also do not 
have the technical training of a cognitive psychologist specializing in the study of 
mind and mental capacities. Instead, flight examiners configure whatever there is 
in the pilots’ environment, internal or external to the cockpit, and observe whatever 
the pilot under examination produces. These external productions are taken as 
documentary evidence for some underlying skill (see Chapter 2). What particular 
skill is at issue depends on the airline and the particular (explicit or implicit) model 
it uses to conceive of and test a pilot’s skills. In this approach, the particular states 
of an instrument or the events outside the cockpit are taken in a scientific sense, as 
objective occurrences (facts). At times, flight examiners do attempt to get into a pilot’s 
head after the fact, during debriefing, for the purpose of arriving at an assessment 
for rating or pass/fail purposes (see Chapters 7 and 10). Failing to see what an 
instrument reads relevant to the present stage of the flight tends to be attributed to 
a lowered or lack of situation awareness. 











One frequently used response model is referred to by the acronym (T-, F-) DODAR, 
which stands for diagnosis, option (generation), decision-making, assignment, and 
review; the modifiers T- and F- that some airlines add to the basic model refer to time and 
fly, respectively. This model can be represented in terms of existing views on situation 
awareness from a self-actional perspective (Figure 1.4). Thus, as soon as a non-normal 
situation announces itself, pilots first ascertain that they have stabilized the aircraft, 
which is achieved by choosing situation-appropriate speed, power, and configuration. 
That is, there is an immediate response prior to making any other decision. Only after 
the aircraft is ascertained to be on a safe trajectory and with safe parameters that pilots 
attend to and bring into play the other aspects of the emergency response. 

The first part of the response model concerns the diagnosis, which is essentially 
accomplished by the three aspects of situation awareness: perception, comprehension, 
and projection (Endsley 1994b). After having diagnosed the situation, the pilots 
generate a number of possible alternatives (options) and assign tasks to be taken. They 
then respond to the situation. The response changes the environment. The decision 
therefore needs to be reviewed to ascertain that the preceding decision still satisfies 
the changing conditions. 
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Figure 1.4 An integration of the (F-) DODAR response model with a theory of situation awareness in 
decision-making and action (response). 


The pilots assessing the performance in the Missed Approach scenario (see box), 
directed so by the task description (which reproduced the everyday assignment of a 
flight examiner), evaluated the two pilots separately scrutinizing the video for evidence. 
They are thereby predisposed to act consistent with a self-actional model. Thus, for 
example, the PF makes a statement about the weather downwind, which is captured by 
the perception component in the model (Figure 1.4). The comment “we have to watch” 
may be taken as a sign of comprehension, perhaps even a certain level of projection. 
However, according to the model, one would now anticipate some discussion to occur 
where different options are deliberated; but there is no evidence in the video clip of such 
a deliberation. It then is understandable why no explicit decision is made, and, simply 
continuing was the decision. If there had been comprehension and projection, the IMC 
situation should not have come as the surprise that it evidently is to the pilots in the 
scenario. In the analysis of the events immediately following the go-around call, analyses 
consistent with the self-actional models focus on what each individual pilot does. Such 
analyses may focus on the evidence consistent with confusion and the initiation of a 
right turn, but may differ on whether the PF was confused about where he was (which 
would be attributed to loss or lack of situation awareness) or whether he did not know 
in which direction to turn (i.e., failed to remember that in a circling approach the turn 
will always be in the direction of the airport). The PM, on the other hand, because he 
corrected the PF, is assessed as having this knowledge or situation awareness. 

With respect to situation awareness, an interesting contradiction arises when 
considering the actions of the PF during the go-around. As shown in Chapter 3, the pilot 
pairs assessing this scenario differed in their ratings of the PF on situation awareness, 
ranging from 1 (very poor) to 3 (satisfactory) generally pointing out the failure to 
plan for the adverse weather earlier perceived and being confused about the direction 
of the turn to be taken. However, whereas there is uncertainty about the direction of 
the turn, the PF pitches the aircraft up. He does so even though the flight director 
bars indicate that a descending flight path is to be flown (Figure 1.5). Here, therefore, 
the pilot is confronted with a contradiction, where the interpretation of the situation 
outside the cockpit requires a go-around decision whereas the interpretation of the 
electronic attitude and director indicator requires continuing on the approach. Just as 
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Figure 1.5 The electronic attitude and director indicator as it would have been just after the PF in the 
Missed Approach scenario announces the go-around. The aircraft is still banked to the left but already 
has the nose pitched up (see aircraft symbol) even though the flight director command bars (magenta) 
suggest flying downward. 


the PF changes the direction of the flight to the left after the PM suggests a correction, 
a strictly information-oriented approach suggests that the attitude should have been 
corrected. The PF flies this aspect of the flight correctly, thereby manifesting situation 
awareness in the face of contradictory input from the environment. Some readers may 
find the actions of the PF self-evident. But it apparently is not, for being confronted 
with the same type of instrument reading, the PF in Lao Air Flight QV 301 did indeed 
follow the flight director instead of extracting the aircraft—with terrible consequences. 

Much of the research and practice concerning what pilots know and how they 
perform—i.e., related to cognition—takes the classical psychological perspective 
outlined here, postulating certain underlying forms of cognition and cognitive skills 
somehow lodged in the brain case of individual pilots. This manifests itself in the 
different typologies of cognitive factors underlying observable performance have been 
proposed and are associated with measurement instruments and behavioral markers 
to establish the level of these factors. These include, for example, (a) the NOTECHS 
approach and its categories of “cooperation”, “leadership and managerial skills”, 
“situation awareness”, and “decision-making” (Flin et al. 2003) and (b) the MAPP 


2 66 


approach with its skills of “situational awareness”, “maintenance of tolerances”, 
pp 
99 ce. 


“decision-making”, “knowledge”, “management”, and “communication” (e.g., Mavin 
et al. 2013). 


INTERACTIONAL MODELS OF PILOT PERFORMANCE 


A smaller number of studies follow situated and distributed cognition approaches. Such 
studies tend to approach the cockpit as the relevant unit of analysis and focus on the 
information exchanged between the different agents, including pilots and the system 
(Bloomberg 2011). Such studies differ from those in the preceding section (which 
take the individual as the unit of analysis) by choosing the interaction between the 
components of a joint cognitive system as the relevant unit within which cognition is 
located. The literature sometimes is referred to by the term distributed cognition; more 
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recently the term joint cognitive system has been used more frequently, especially in 
studies concerned with cognitive engineering. 


Fundamental Assumptions 


The question about where to locate cognition is not concerned with conscious awareness 
but with where relevant information is located, stored, or processed. This may occur 
elsewhere than in the pilots’ mind, such as when pilots calculate takeoff and landing 
speeds and then write the results with a grease pencil on the laminated pages of a speed 
card booklet. This booklet is the memory device and the recording is the associated 
representation. The pilots themselves do not have to store the speeds in their intermediate 
or long-term memories. When the speed is needed, pilots simply go to the booklet and 
get the relevant information. Similarly, there are speed bugs on the air speed indicator 
that can be set so that specific speeds do not have to be kept in the pilots’ mind. An 
appropriate unit for understanding the calculation, memory, and forgetting of speeds, 
therefore, is the cockpit as a whole. The approach is explicitly cognitive in the sense 
that it answers questions about “how information is represented and how representations 
are transformed and propagated in the performance of tasks” (Hutchins 1995b). It turns 
out that such an orientation leads to interactions rather than individual actions as the 
locus of the significant dimensions of relevant cognitive phenomena. 


Joint Cognitive Systems 


In aviation, two studies in particular provide detailed analyses of flying aircraft from a 
joint cognitive systems (distributed cognition) point of view. Both investigate speed- 
related events, but re-orient research by changing the research question: “How a Cockpit 
Remembers Its Speed” (Hutchins 1995b) and “How a Cockpit Calculates Its Speeds” 
(Henriqson et al. 2011). As the paper titles suggest, the question is not how the pilots 
individually or collectively calculate and remember speeds but instead how the cockpit 
as a whole does its speed-related work. Not only pilots are agents; the system, too, is 
an agent. It provides pilots with information: It tells them that something is wrong, 
which they would not have looked for unless they were being told. The question is to 
investigate how something like speed is represented and processed within the cockpit. 
Both representing and processing, though it involves pilots, also occur outside the pilots. 
Whatever happens in the mind of the pilots, their mental memories, is considered as 
“additional media in the cockpit system” (Hutchins 1995b) not as the primary means, as 
the self-actional models would like us to believe. With respect to the speed investigated 
in that study, the representational devices include the speed card booklet, the airspeed 
indicators with their internal and external bugs, the speed-related verbal exchanges 
between the pilots, and the gross weight display. 

It might be assumed that calculations can only occur in the mind of the pilots or 
the calculators. But this is not so. For example, for a given gross weight, the speed 
card booklet provides pilots with the appropriate speed. That is, the calculation has 
already been done and is embodied in the structure of the booklet so that the result 
is accessible to the pilots when they need it. Selecting the correct weight inherently 
leads to selecting the correct speed. Whereas these representations and calculations are 
durable, inscribed in the medium, verbal representations (i.e., pilots’ spoken words) 
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are ephemeral. In some circumstances—see the accident of the Ansett DHC-8-102 
near Palmerston North, New Zealand, where the first officer forgot information he 
had received only a minute earlier (see box, Chapter 8)—pilots may forget something 
that has clearly been said and repeated; and they might have forgotten even though 
they have repeated the information back to the original speaker (e.g., control tower). 
Moreover, some information is redundantly represented, such as the current speed, 
which is available on the two airspeed indicators, or the colored speed bugs, which 
are set the same on the two instruments for PF and PM. The up trim of the live engine 
when a second engine fails also is available in multiple representations (i.e., torque = 
100% and lit UP TRIM signal light), a situation that becomes an issue in the teaching 
for situation awareness presented in Chapter 10. 

The joint cognitive system approach focuses on functional systems. Thus, the 
speed bugs on the airspeed indicator are parts of a functional system, a constellation 
of structures some of which are external to pilots and others internal. It is their 
relation—shaped by the exchanged representations—that makes for the cognition of 
the cockpit. For example, in the calculation of speed, the particular aircraft investigated 
provides the gross weight of the aircraft on the fuel quantity indicator. Taking this 
representation to the speed card booklet yields a specific landing speed as the result of 
the implicit calculation. This representation then is made perceivable in the cockpit in 
two forms: verbally, in the announcement of the speed on the part of the PM, and in 
the relevant setting of the speed bug on both speed indicators. These bugs constitute 
the external memory of the speed. The pilots do not have to remember it because the 
bugs do it for them. When required during the landing approach, the pilots can access 
the representation in the cockpit. This entire process may be called “a cockpit system’s 
‘memory’ because it consists of the creation, inside the system, of a representational 
state that is then saved and used to organize subsequent activities” (Hutchins 1995b). 

In this approach, pilots come to be considered differently than in the traditional 
self-actional approaches, where phenomena such as memory are thought of as higher 
psychological functions internal to the pilots. In the cockpit, the pilots are part of a 
functional system that calculates and remembers. Cockpit memory exists in the ensemble 
of interactions of all parts. To the pilots, the system attributes to pilots tasks that 
consist of some “combination of recognition, recall, pattern matching, cross modality 
consistency checking construction and reconstruction that is conducted in interaction 
with a rich set of representational structures, many of which permit, but do not demand, 
the reconstruction of some internal representation” (Hutchins 1995b). Readers clearly 
see how this view differs from the self-actional approach presented in the preceding 
section, where everything falling into the cognitive category is made to occur in the 
mind of individual pilots. 

In the joint cognitive systems approach, agents and artifacts interact, where agents 
are taken to be human or machine (Henriqson et al. 2011). Some of the machine agents 
involved in the calculation of speeds include the flight management computer and 
the laptop. They are considered agents because of their contributions to the cognitive 
functions in the cockpit system as a whole. The flight management computer and the 
global navigation satellite system are agents that are given and receive information 
(programming) and that in turn provide the pilots with information. Some pilots 
therefore refer to these as additional pilots that are part of the work of getting the job 
done. Those jobs require interactions between different artifacts, between pilots and 
artifacts, and between the pilots. The consideration of the cockpit as a joined cognitive 
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system makes visible that the system performance is not uniquely a consequence of 
the pilots’ capacities or uniquely the direct result of the way in which the computers 
(computer programs) are designed. Instead, the interaction of people and artifacts creates 
a whole new cognitive system. However, this approach does not require us to consider 
the agents in terms of the system as a whole. Instead, there is an interaction of agents 
that can be defined sui generis. Thus, for example, there are three interacting cognitive 
systems when a pilot reads weights and another enters them into the laptop: a first pilot, 
who reads the load sheet values; a second pilot, who enters the values into the laptop; 
and an information transfer of symbolic values between pilots (Henriqson et al. 2011). 
Nothing prohibits researchers to assume the two pilot-related systems as complete. To 
get the whole system, however, requires analyzing the interactions. 

One analysis conducted in the spirit of joint cognitive systems focuses on the 
practice of mixed-fleet flying generally and flying alternate versions of the same aircraft 
specifically (Soo et al. 2016). After considering the different errors that occur when 
pilots move from a non-glass to a glass cockpit version of the same aircraft, the study 
concludes that the two situations constitute different joint cognitive systems.' From an 
analysis of the system as a whole, the joint cognitive systems approach allows predicting 
where errors will occur that are impossible to locate when the pilots were considered 
on their own. These authors showed, in the case of pilots who returned from flying a 
glass cockpit to an analog cockpit, how a great number of errors occurred. As a direct 
result of another study of the errors that may arise in mixed-fleet flying, the airline 
where these studies were conducted abandoned the idea of mixed-fleet flying altogether, 
establishing instead two parallel fleets operating in parallel over a transition period 
(Mavin et al. 2015). The airline abandoned the idea of mixed-fleet flying even though 
the manufacturer has created the new version of the aircraft such that pilots type-rated 
on one may fly both types. Errors do occur, however, when pilots move between two 
aircraft with traditional and glass cockpit, a fact that can be taken as implied in the 
conclusions of the final report on a fatal flight of Lao Air where mixed fleet flying was 
practiced. The report states: 


To ensure that its flight crew are competent in operating the conventional ATR- 
72 aircraft and glass cockpit ATR-72 aircraft after relevant training. One of the 
objectives of the transition course from the basic ATR72 to the ATR72-600 is to 
give pilots new references and to allow them to re-work their visual scan and the 
callouts linked to awareness of automated modes. (LPDR 2014) 


In this report, the blame is distributed between airlines and pilots rather than to the 
systems involved. The pilots should have had references, and the airline should have 
better prepared the pilots. One study suggests that such errors occur in part because the 
pilot bodies do not forget so that their flows (kinetic melodies) remain intact, at least 
initially, immediately after moving between the different cockpits (Roth et al. 2015). 
Individual pilots are also blamed in the course of the present set of investigations. As 
the research reported in Chapter 3 shows, most pilots watching the Missed Approach 
scenario (see box) in one way or another blame the PF for the “untidy” or “messy” 
procedure. The PM receives much higher overall scores than the PF whether the 





' In glass cockpits, what used to be analog flight instrument displays are implemented electronically 
(digitally) in the form of LCD screens. 
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assessments are done with or without an explicit human factors model and associated 
assessment instrument (see Figure 3.2). 

An analysis from an interactional perspective, however, not only pursues different 
motives but also may come to different conclusions. Consider two aspects of the Missed 
Approach scenario flight right at the go-around decision: autopilot and go-around 
buttons (Table 1.2). From the joint cognitive system perspective, there are three agents: 
PF, PM, and automated flight control system (AFCS). In the scenario, the PF disconnects 
the autopilot by pressing a button on the control column. The PM inherently is not 
aware of the pressing, but the AFCS is. In response to the autopilot disconnect, the 
AFCS produces two responses that make available information about the change in 
state. First, there is an aural warning in the form of a cavalry charge; it is penetrating 
enough so that it cannot be overheard (though we do not know whether the information 
is comprehended or processed). Second, the light next to the autopilot button on the 
AFCS control panel goes off; from the reactions of the scenario pilots, it cannot be 
derived whether they are or are not aware of this light. The PM then reconnects the 
autopilot. Because the hand has to move up to the AFCS panel, the action inherently 
is available to the perception of the PF (Nevile 2004a), though it remains unknown 
whether the PF actually perceives (is conscious of) the hand movement. But the AFCS 
is “aware” of the action, which it announces in two ways: the light next to the AP button 
comes on and the aural warning stops. In this instance, information is available in the 
form of multiple representations and, therefore, produces a level of redundancy that 
contributes to safeguarding the system as a whole. Now consider a second situation. 

In the case of the go-around button, the same level of redundancy does not exist 
(Table 1.2). This leads to contradictory information within the system, which is expressed 
on the electronic attitude and director indicator (Figure 1.5). In the scenario, the PF 


Table 1.2 Actions and Information Available to the Different Agents—Pilot Flying (PF), Pilot 



































Monitoring (PM), and Automatic Flight Control System (AFCS). 
Agent Action Noticed by (mode) 
PF PM AFCS 
Situation 1: Autopilot (AP) 
PF AP disconnect YES NO YES 
AFCS AP disconnect aural warning YES YES AURAL 
AP disconnect visual (AFCS ? ? AP OFF 
control panel) 
PM AP connect (see?) YES YES 
AFCS AP disconnect aural warning stops (HEAR) (HEAR) YES 
AP connect visual (AFCS control (see?) (see?) AP ON 
panel) 
Situation 2: Go-around (GA) button 
PF GA not pushed ? ? YES 
Machine Flight director down ? ? (EADI) 
PF Nose up ? ? (EADI) 
YES (feel) YES 
(feel) 























Note: EADI = electronic attitude and director indicator 
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does not push the GA button on the power lever, as the standard operating procedures 
require. The PM certainly cannot be aware of the PF’s action (pushing or not pushing). 
As far as the AFCS is concerned, the button is not pressed. Therefore, it does not switch 
into the go-around mode, which would have included a change in the director bars on 
the electronic attitude and director indicator. Instead, the AFCS continues to direct the 
PF to descend (Table 1.2). Contrary to the flight director bars, the PF in the Missed 
Approach scenario pitches the aircraft up. Both pilots can feel the result, and there 
is similar input to the AFCS, which displays the aircraft symbol as heading upward 
(Figure 1.5). The information now displayed is contradictory. Indeed, the director bars 
still point down when the PM announces positive rate of climb, clearly available on 
the traffic advisory vertical speed indicator (TCAS VERTICAL) and on the increasing 
tendency on the altimeter. As seen in Chapter 3 (Figure 3.6), some flight examiners 
explicitly talk about this contradiction and what would have happened if the PF actually 
had followed what the flight director suggests. 


TRANSACTIONAL MODELS OF PILOT PERFORMANCE 


Transactional approaches to cognition and performance emphasize the irreducibility of 
actions to single individuals (pilots) and approach piloting from a systems perspective 
where the whole is not composed of and reducible to elements. There is therefore no 
interaction between self-sufficient cognitive systems. Instead, there are constitutive 
part-whole relations so that any part implies all other parts. In other words, no part 
of the system can be understood on its own. The situation described in the preceding 
section, where two pilots move from a glass cockpit to a traditional analog cockpit, 
therefore involves two different systems. Thus, even the pilots cannot be considered 
to be the same because different actions are solicited from them by the two systems 
considered in their entirety. The view therefore is decidedly different from the different 
interactional takes, which either put the system together from self-sufficient parts or 
focus on the information moved rather than on the actions of individual components. 


Fundamental Assumptions 


Transactional theories take the view that higher psychological functions and cognitive 
processes are societal and historical through and through and that “the structure of 
mental activity—not just the specific content but also the general forms basic to all 
cognitive processes—change[s] in the course of historical development” (Luria 1976). 
Consciousness is taken to be the most advanced form in which reality is reflected. 
This consciousness continuously changes, is produced and shaped in culturally and 
historically specific activities, and not only orients and adapts itself to the environment 
but actively shapes it. For example, when the pilot flying in Scenario 1.2 below says, 
“Going IMC” (turn 01) to which the pilot monitoring reacts and upon which he acts, 
then the words constitute the consciousness of both; and this consciousness shapes the 
environment in and upon which the pilots act. 

Analyses that take into account the transactional nature of life typically take the 
interdependence of talk and action as their starting point (e.g., Nevile 2004b). Thus, even 
if there is just one pilot talking, the talk and non-talk activities are oriented to take into 
account the perceptual needs of the other pilot. That is, whatever a pilot says and does 
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is produced for the other as much as it is for the speaker; talk and action are recipient- 
designed and therefore include characteristics of both producer and receiver. Consider 
the Scenario 1.1 (Figure 1.6) of talk and action that occurs at the moment when a pilot 
monitoring (PM) checks for and announces an engine flameout (Chapter 10, Fragment 
10.3). As the talk unfolds, the PM reaches forward such that the hand can be seen to be in 
the direction of the torque gauges. Both the talk and the reaching forward are produced 
for the pilot flying (PF), that is, they are also produced in-order-to be seen and heard 
(Nevile 2004a). Thus, when the PF says, “confirmed”, then “engine flame-out”, for the 
purpose of the events from the perspective of the cockpit, is the consciousness about the 
current state of affairs. Moreover, the movement of the hand toward the torque gauges 
manifests that these rather than other instruments have been checked. 

To decrease the possibility for misunderstandings, the Federal Aviation 
Administration instituted the sterile cockpit rule, according to which crewmembers are 
to refrain from all non-essential doings during critical phases of the flight (FAA 2015). 
Here, the aircraft is just taking off and immediately prior to Scenario 1.1 (Figure 1.6), 
gear up has been called and executed when the flameout occurs. If non-essential talk were 
to occur—as when the PF in the Missed Approach scenario (see box) apologizes for the 
turn confusion in the middle of the go-around procedure and before acceleration altitude 
has been reached—this is treated as a serious breach of standard operating procedures. 

The two seminal papers discussed in the preceding section investigate cognition 
involved in navigating modern, multi-crew aircraft contribute to the development of 
a joint cognitive system approach. Although the joint cognitive systems approach 
constitutes an advance over more traditional approaches focusing on individual pilot 
performances, both papers fall short of describing cognition sufficiently. Based on 
extensive ethnographic data—including recorded modified think aloud protocols, 
debriefing sessions following (recorded) simulator bi-annual simulator assessments, and 
simulator performances followed by stimulated recall using simulator recordings—our 
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Figure 1.6 Scenario 1.1: Two pilots deal with an emergency situation, an engine flameout. (Transcription 
conventions in Appendix A.) 
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own work shows that a considerable amount of flying involves bodily and embodied 
knowledge sequentially and temporally organized in situation-specific kinetic melodies 
(Roth et al. 2015). These become apparent when pilots are asked to fly multiple 
aircraft (mixed fleet flying), where the flow patterns from one cockpit are triggered in 
a physically and cognitively different cockpit. 

Thinking about pilot performance in terms of kinetic melodies reorients 
investigators to movement and flow rather than to static bits of information passed 
around. Just as a symphony requires the coordinated movements of multiple instruments 
and instrument groups, the performance of a cockpit requires the coordination of its 
different constituent parts: automation, instruments, and pilots. Each plays a part in 
the symphony, like the melodies of different instruments. If the different lines and 
melodies are not coordinated, cacophony will result—a little like hearing the independent 
warming up of orchestra members. But the coordination is not caused by individual 
actors somehow cognizing and acting upon an external structure; nor does the orchestra 
as a whole or its conductor cause the behavior of individual players. The individual 
musician and the orchestra are paired entities. In other words, the behavior of each is 
implicated in the behavior of the other. When the go-around procedure is flown well, 
all flows (two pilots, instrumentation and automation) are coordinated leading to the 
cockpit equivalent of a well-sounding symphony (Figure 1.7). When one of those parts 
begins to play a different note, the whole symphony might fall apart. Observers of the 
Missed Approach scenario notice such a falling apart; and they describe the performance 
as “messy” or “untidy”. 




















“Positive rate” 
“Flaps 15, power set” 








“Go around, set power, flaps 15” 


Figure 1.7 Inakinetic approach, performance is understood in terms of kinesthetic (or kinetic) melodies. 
The go-around procedure requires the coordination of three flows (lines of flight), one for each pilot 
and one for the technical system. These flows need to be coordinated like the melodies of different 
instruments in a band or orchestra. 
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In the Missed Approach scenario, the PF announces a go-around and moves power 
levers forward; he disengages the autopilot (not commanded by standard operating 
procedures) but does not push the go-around button as stated in the standard operating 
procedure. It is as if someone in the symphony played a series of wrong notes or left 
out some notes. The autopilot disarmed aural warning sounds (cavalry charge), and the 
hand of the PM moves around over automated flight control system panel then pushes 
the “AP” button thereby turns the autopilot back on. None of this is contained in the 
description of the go-around found in the standard operating procedures. The PM is not 
just playing out a melody but, in performing, adjusts to the state of the system (cavalry 
charge) to return it to the state where it is supposed to be. On the other hand, when the 
standard operating procedure (1.e., “musical score”) standard call for heading and speed 
to be checked and announced does not come forth (as the procedures require the PM), 
the PF “picks up” and announces it (i.e., plays this part of the tune). As a result, even 
though there is trouble and confusion, and even though there are uncalled-for events, 
the aircraft ultimately is extracted successfully from the situation. Just as mis-takes 
in improvisational jazz may offer opportunities for a new, endogenously produced 
organization and therefore constitute new takes (Klemp et al. 2008). In other situations, 
mistakes are such that they do not support recovery of any kind, such as in the case of 
the fateful TransAsia GE235 flight (February 4, 2015), where the overall composition 
does not come together ending in tragedy (see box, Chapter 10). 


An Experience and Lived-Experiencing 


One transactional model has been conceptualized with the category experience, where 
“category” is understood to mean that the term captures the worldly phenomenon as a 
whole. When other theoretical approaches use the term category, then they refer to one 
of many components of cognition, such as in the models of pilot performance, which 
include situation awareness, management, and decision-making as basic categories. 
When experience is used as a category in transactional models, it refers to the unity/ 
identity of the person and environment, which leads to a unit that might be designated 
by the term person-acting-in-environment-as-a-whole (Dewey 1934/2008; Vygotsky 
2001). Here, “experience” should be understood not as something completed, not as 
“an experience” but as “lived-experiencing”, which is a more appropriate rendering of 
the German Erlebnis and the equivalent Russian perezivanie used in philosophy and 
psychology. This is especially important because the English experience also is used 
to translate a very different concept into English, opyt (Rus.) and Erfahrung (Ger.), for 
which the most appropriate English translation is “an experience”. Lived-experiencing is 
not something punctual but has duration because there is a past and future that belongs 
to every now; and any lived-experiencing is connected into a changing material context 
(Schiitz 1932). This means that persons and their identities, knowledge, and other 
characteristics cannot be determined independent of the environment or independent 
of time; and the relevant characteristics of the environment cannot be determined 
independent of the individual. That is, the role of environmental characteristics in 
performance is a function of the person; and the role of the person characteristics in 
performance is a function of the environment. Moreover, neither person nor environment 
are constant but continuously change. Consider the following excerpt from the Missed 
Approach scenario, which happens at the point when the aircraft enters the rain cloud. 
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Scenario 1.2 


01 PF: Going IMC. 

02 PM: Rain shower there. 

03 PF : Missed approach was right hand, wasn’t it? 
04 PM: Negative. Left hand! Left hand! 

05 PF: Itisleft? 


The phrase “Going IMC” does not have its causal origin in the brain or in 
representations (Il’enkov 1977). The brain was already active when light from the 
environment impinged on the eyes of the pilot. The pilot actively attended to, was 
in tune with, the environment. But the particular state of the environment is not the 
result of the willy-nilly construction of an autonomous brain. “Going IMC” is a reply 
that has been shaping with the perceptual process itself. The brain has not stayed still, 
waited until some information package has arrived, which, depending on the use of 
information processing or constructivist theory, then is taken as is or interpreted so 
that a reply could be decided upon, planned, and then delivered in the form of a verbal 
description. In self-actional approaches, the rain shower or IMC has to be represented 
and entered into the short-term memory to be acted upon. However, as a close analysis 
of everyday actions in the example of a car turn signal shows, the driver does not 
experience such processing; the information does not somehow stand between the turn 
signal and the mind of the driver (Heidegger 1977). Instead, the driver acts upon the 
turn signal directly. Some readers may find this counterintuitive given what appears 
to be instantaneous nature of visual perception. Consider therefore part of the verbal 
exchange re-transcribed to take into account the perceptual part. 


Scenario 1.3 


(03 PF: (says) Missed approach] 


was right hand wasn’t it? 

















04 PM: (hears) Missed | (says) Negative. Left hand! 

approach was right hand Left hand! 

[wasn’t it? 

05 PF: (hears) Negative. Left hand! (says) It is left? 
Left hand! 








In this transcription, those parts are shaded grey where someone is active in ways 
that are not normally transcribed. Understanding the origin of the boxed phrase that the 
PM produces requires attending to what the PF has said (turn 03), to the environmental 
conditions, and to the current state of the aircraft. If anything, the PM’s statement 
(turn 04) is the result of the effect that the natural and social environment and the 
activity have on him. The PM cannot know what is coming at him, what the voice of 
the PF will have said when it comes to an end, that is, what the PF says in a future 
perfect sense. The PM is characterized by lived-experiencing of what is happening, 
because it is still unfolding, and because nobody knows what kind of event will have 
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materialized in the end. This means that the PM has to open up to be affected. But this 
opening up, the active listening to another, orients itself in particular ways. Listening 
is not the same across situations: we may listen for the content, to be able to repeat, 
to find out an attitude, to show that we are listening, or for understanding when it is 
our turn (e.g., Sudnow 1979). As a result of these orientations, our perceptions differ. 
In taking a look at the watch to see how much time has passed since we started taking 
time, we may completely fail to see what time it is (Roth 2012). Thus, hearing and 
perceiving are relational phenomena, irreducibly binding the pilot-perceiving-the-rain 
into a single and irreducible unit. 

In turn, the PM has an effect on the social environment; but this effect is available 
only with the reply on the part of the PF to what the PM says: “It is left?” If, for the 
moment, we consider that there are things happening in the brain of the PM associated 
with the speech act “Negative. Left hand! Left hand!” then it has to begin with the 
active attention that he gives to the PF. As the PM attends to the speech of the PF, 
the brain can be assumed to be working and already setting up a reply, which in this 
instance comes without delay that would have been required if the brain had to interpret 
the phrase as a whole. It is only as a whole that it could be interpreted as a question, 
statement, request, invitation, and so on. But the PM cannot know what he has done, 
how he has affected the PF until he hears the reply. That is, the thinking associated with 
the boxed act of speech—technically referred to as to locutionary act as distinct from 
illocutionary (intended) acts and perlocutionary acts (effects)—extends at least from 
the instant that the PF begins to speak (turn 03) to when the PF has ended speaking 
in turn 05. We do not have access, however, to what happens in the brain of the PM. 
Indeed, the PM can know his thinking only through his speech (Vygotsky 1987). As 
the phenomenon of having a sudden realization shows, the brains have done a lot of 
work without the persons knowing thereof when they find themselves in the situation 
of just having had an insight. That is, even for the PM his thought does not express but 
completes itself in speech. As a result, “the [phrase] absorbs intellectual and affective 
content from the entire context in which it is intertwined. It begins to mean both more 
and less than it does when we view it in isolation”. The phrase says more than when 
viewed on its own because it accounts for the specific context, making some aspect 
of it the currently accented visible. It also moves the situation ahead in that it has an 
effect on the surroundings rather than constituting some ephemeral and non- (meta-) 
physical “meaning” in the equally ephemeral mind of the PM. The phrase also says 
less, because of all the different ways in which it can be heard in different settings, only 
one way has been bearing on the current event. 

It is apparent that any thinking associated with Scenario 1.3 changes as time 
unfolds from left to right. Thinking is affected both from the outside, what is given to 
and received by the person, and by the activity of the brain, such as when the outcomes 
of its own activity are available at the end of speaking as completed thought. Inner 
speech is some intermediate station between brain activity inaccessible to the person 
and his speaking. Thinking is further affected as the PM attends to the speech of the 
PF that ensues. Rather than considering thought in itself, in the way this is done in 
much of the work on cognition, it is better to think of it in the service of the person. 
Once thinking is considered in this way, we realize that “any thought has movement. It 
unfolds. It fulfills some function or resolves some task” (Vygotsky 1987). Such tasks are 
social and situated through and through, just as is the specific uses of words in specific 
situations. It does not help us knowing what a word “means” in general to understand 
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its specific function in the situation—which may indeed be completely independent 
of the semantic dimensions of the word (e.g., Roth 2015c). Moreover, the ability to 
recognize that the semantic dimensions of a word are irrelevant to the situation (e.g., 
because everything to be communicated is presented in the prosody) never is included 
in theories but requires a sense for the social game that is currently in play. 


Cultural-Historical Activity Theory 


An important theoretical framework in the understanding of work and (cognitive) 
ergonomics is cultural-historical activity theory (e.g., Bedny and Karwowski 2004). It 
was proven useful in transportation studies (e.g., Decortis et al. 2000) and in simulation 
and training in high-risk industries, including the operation of nuclear reactors, surgery, 
and military operations (De Keyser and Samurgay 1998). As elaborated in Chapter 8, 
it also is a useful framework of debriefing in the context of simulation because of its 
particular attention to the functions of tools and division of labor in human production. 

The fundamental starting point of cultural-historical activity theory is the idea 
that anything that makes humans different from animals is due to human society 
and the human society-specific (i.e., societal) relations that continuously produce 
it. Universally valid ideas and concepts, such as the (monetary) exchange-value of 
goods, are expressions of relations between people reflected in the relations between 
material things that are attributed to some of these same material things; these ideal 
expressions, including scientific concepts or money, only arise with sufficiently large 
populations (Il'enkov 2012; Marx and Engels 1962). To understand individual human 
behavior requires understanding society as a whole. There is therefore a primacy of the 
societal; and sociogenesis is the process by means of which individual behavior has its 
genetic origin in the relations with other people. As a result, every higher psychological 
function was a relation with another person before becoming a function of the individual 
(Vygotsky 1989). This has led to the category of activity, which is conceived of as the 
smallest unit that preserves all characteristics of society (Leont’ev 1978). Society is a 
system of activities that produces everything required to meet human needs, so that even 
consumption (leisure) is theorized as activity. Typical activities include manufacture 
(e.g., aircraft) or farming (e.g., vegetable or grain). In the service industries, such as 
aviation, the transportation of people and goods from some location A to some location 
B is the outcome of the relevant activity system. Most importantly, cultural-historical 
activity theory really focuses on movement and transformation and, therefore, theorizes 
the entire production process within the same category and unit of analysis. An activity 
lasts over a period of time; and, thus, the category encompasses change of the whole 
and of its parts. We therefore must not think of a pilot or some cockpit artifact and 
tool as a fixed part in activity. Instead, the minimum unit of cultural-historical activity 
theory is a unit of change. That is, if we look at an activity at different points in time 
between the beginning and the end, then the parts will be different. These differences 
are internal to the category, which, therefore, manifests itself in different way even 
though conceived as one unit. The following analogy may help: 

Consider the transformation of a square into a parallelogram. In most theoretical 
approaches, the basic unit would be the rectangle, which is transformed by an outside 
force F (Figure 1.8a). Cause (force, agent) and effect (transformation) are cleanly 
separated. In the activity theoretic approach, the entire production is a unit of analysis 
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unit 





unit 


Figure 1.8 Examples of units of analysis. (a) The unit of analysis is the square, which is changed by an 
outside force, a pattern of thinking underlying traditional cause-effect studies. (b) In cultural-historical 
activity and other transactional theories, the force is inside the system, which therefore is self-moving; 
no cause-effect reasoning is possible, for the effect (transformation) is used to determine the force(s) 
at work (F) as much as the force(s) are used to determine the effect. 


(Figure 1.8b). Because a unit inherently cannot be broken down further without the loss 
of sense, the force and transformation no longer are separable and independent. Force 
and transformation mutually imply one another such that the specific forces that are 
said to have caused a transformation always are a posteriori attributions. Contradiction 
therefore is inherent in the system because anything we may observe—the square, 
parallelogram, or any intermediate stage—is a manifestation of one and the same 
system. This is like what we see in quantum mechanics where we may no longer talk 
about a system abstractly (e.g., is light a wave or is light a particle?) but only in terms of 
concrete observations (1.e., it depends when, where, and how we look). Time, therefore, 
also is inherent in the living system, which undergoes continuous transformation (if 
not, it would be dead). 

Activity is not constant but changes in the course of its history and differs across 
societies (cultures); this leads to the adjectival modification of the theory as cultural 
or societal (in German and Russian publications) and historical. It is quite apparent 
that farming grain differs according to the type of grain, the available (cultural) tools, 
the stage of societal development, and so forth (Roth and Lee 2007); very different 
practices and cultural cognition are involved whether rice or teffis grown and harvested. 
Whereas flying an aircraft from airport A to airport B may be abstractly conceived “to 
be the same” independent of aircraft, it is sufficiently different from the perspective 
of pilot cognitive practice to warrant different type-rating requirements. In other 
words, the practice of type rating and limiting commercial pilots to flying only those 
aircraft for which they have undergone type rating is a manifestation of the difference 
in cognitive practice. Activity, behavior, and consciousness differ when (history) and 
where (country) we investigate what happens in the cockpit. Different cultural patterns 
appear to be clearly at work in cockpit communication when Asian cockpit crews are 
compared with Western crews. 

Any activity, such as getting an aircraft from airport A to airport B is a system 
with parts broadly conceptualized under the headings of subject (individual or group), 
object, tool, rules, community, division of labor, and outcome (Figure 1.9). These 
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parts are not independent and, therefore, do not constitute elements from which the 
activity system can be constructed. Instead, the very nature of any material and social 
phenomenon is a function of the system as a whole. One of the classical examples 
illustrating the difference is that of hammering (Heidegger 1977). The situational totality 
and hammering are discovered before (and take precedence over) any individual useful 
thing and its objective characteristics. The material object here is the aircraft together 
with its load (passenger, cargo) and fuel at airport A. The motive is the transformation 
into the future state, the aircraft at airport B. In the course of the activity, the object 
is transformed into what initially merely is the (ideal) motive, whereby the object is 
transformed such that it gets closer and closer to the state conceived of and anticipated 
in the motive. Activity theorists therefore often speak of the object/motive of activity. 
This object/motive is of (conscious) societal nature because it captures generalized 
need satisfaction (getting cargos and passengers from A to B). Any work is shaped 
by the function that the thing produced has within the other activity systems where 
it is consumed (Schütz 1932)—e.g., the pilots’ work outcomes have a function in the 
leisure or business activity of the traveler. Within this unit, there is continuous change: 
fuel is used, humans get exhausted, engines fatigue, and so on. These changes are the 
counterpart of production: consumption (Figure 1.9). 

Movements within activity are thought of in terms of levels of inclusiveness. At 
the highest level, there is the object/motive satisfying a societal need. An activity is 
realized by conscious actions produced in the pursuit of conscious goals, the selection 
and nature of which depends on activity. There is therefore a constitutive relation 
between the two. Thus, the same action (e.g., reading a manual or a quick reference 
handbook) may serve different functions in different activity systems so that the 
specific sense of an action is a function of the activity, which only exists because of 
its realization through concrete actions. Research shows that engineers in the field use 
the same mathematical equation very differently than physicists in their laboratory; 
and the same design drawings of an aircraft have very different cognitive functions 
depending on the department of the manufacturer where they are used. Each action of an 
individual has a conscious goal. Conscious goals and actions, however, consist of series 
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Rules Community Division of Labor 

- standard operating - ground crew - Captain 
procedures - management - first officer 

- safety first - dispatchers - cabin crew 


- crew schedulers 
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Figure 1.9 Related to aviation, the activity system is constituted by the entire transformation of the 
object (aircraft at A) into the outcome (aircraft at B). The system and its parts are mutually constitutive; 
and no part can be understood independently of all the other parts. 
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of movements of a shorter duration. These movements are conceptualized as operations 
and are not present in consciousness. Concrete operations produce actions; but they 
are executed only because the presence of goal and action requires them. Any complex 
action—such as shifting gears in a manually shifting car or the go-around following a 
missed-approach declaration—consists of operations that string together like the notes 
in a melody (see Chapter 9). The goal is the trigger for the melody to unfold on its 
own and without requiring further conscious awareness and attention to the individual 
operations (notes). At the lowest level, there are brain events that are constitutive parts 
of operations. It is not that the brain events produce the behavior. Instead, as shown in 
the case of typewriting above, the shape of the keyboard historically and situationally 
conditions the brain events of the person using it. 

Any change, such as when a cockpit with analogue instruments is redesigned and 
transformed into a fully electronic (glass) cockpit, constitutes a change of the activity 
system as a whole. This is apparent from the representation of the system, which changes 
in its entirety when one part is changed because the alteration “ripples” throughout 
the system. Because changes associated in the instrumentation itself in the transition 
from a cockpit from its analog to digital (glass cockpit) version also involve changes 
in the standard operating procedure (rules, Figure 1.9), problems and failures can be 
anticipated when people, with dispositions and repertoires of habit, move from one to 
the other system. The more durable habits and dispositions (bodily memories) carry 
over such that the system as a whole functions at a suboptimal level. This is shown in 
the results of studies investigating mixed-fleet flying (e.g., Soo et al. 2016). 

Each activity comes with its own particular language often mistakenly referred to by 
outsiders as “jargon”. Even though outsiders may not understand just what the insiders 
are doing with language, the fact that each discipline has developed a subject-specific 
language (discourse) that allows members to talk shop in an efficient manner should 
indicate that “jargon” is highly purposeful and effective. The language and all the other 
aspects of an activity are highly integrated to the point that they form a single unit. 
This unit, in pragmatic approaches to language, goes under the label of language-game 
(Wittgenstein 1953/1997). Indeed, language and activity are so tightly interwoven that 
all distinctions between knowing the language and knowing one’s way around the world 
are erased. This leads to the fact that we may know all the words of a language but still 
not be able to follow what it is that practitioners talk about. Moreover, practitioners 
themselves might come to wonder how talk in a debriefing room has anything to do 
with their work in the cockpit (e.g., “Conceptual Models”, Chapter 8). Consider the two 
pilots in the Missed Approach scenario engaged in the following exchange. 


Scenario 1.4 


01 PF : Going IMC. 

02 PM: Rain shower there. 

03 PF : Missed approach was right hand, wasn’t it? 
04 PM: Negative. Left hand! Left hand! 

O SPF:  Itis left? 

06 PM: Left hand. 


In this scenario, two pilots talk while seated in the cockpit, dealing with the sudden 
emergence of a problem that forces them to call a missed approach and to conduct a 
go-around procedure. It is talk in the world of practice, a first-level order referred to 
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as mimesis, (see Chapter 6). That same situation appears very differently when talked 
about and recorded as part of an assessment. The difference is apparent when Scenario 
1.5 is considered, which represents how a typical flight examiner talks about the same 
situation. In Chapter 6, this type of talk is attributed to the world of text characterized 
by a second-level order denoted as mimesis,. 


Scenario 1.5 


FE : This pilot has lost situation awareness: he doesn’t know whether to turn right 
or left. The pilot monitoring is picking up the error, or they would have crashed 
into the mountain. But the communication was good. Because the PF was 
verbalizing what he was thinking, the PM could pick up and correct the flight 
direction. Situation awareness is the key here. Because the knowledge of the 
situation and the pressure on the PM, the basics disappeared for the PF. This 
is why his situation awareness went out the bottom and the PF was turning to 
the right instead of turning to the left. It was the PM’s knowledge and situation 
awareness that corrected the PF. 


The flight examiner talk in this scenario includes terms that are common whenever 
flight examiners talk about pilot performances, for example, situation awareness, 
communication, and knowledge. We must not conclude all too quickly, however, that 
the flight examiners are using these words in the same way that research scientists do 
when publishing in scientific journals. Although the words are the same, they may 
be used differently, associated with different implicit or explicit theories about pilot 
cognition and learning, and so on. Thus, when considering what flight examiners say 
in the field, we need to understand the special functions and contextual constraints of 
the words-in-use, particularly with respect to the folk theories and folk models that are 
employed as part of mundane non-scientific work. 


FOLK MODELS 


Human lives are characterized by the common and shared ways of knowing how the 
world works. Cognitive anthropologists have come to refer to these ways of knowing as 
folk models or folk theories. These models may be explicit, when people directly draw 
on sets of concepts to explain why things are the way they are, or implicit, where they 
act in patterned and knowledgeable ways as if there was a specific underlying theory. 
Well-documented examples of folk models and theories exist related to physics. For 
example, many people think (believe) that when a ball is thrown, the hand and arm 
transfer something like a motive force to the ball, which keeps the ball going in the 
direction thrown at least for a while. This is the classic idea of an impetus or motive 
force. The theory tends to be implicit, for few if anyone interviewed in research on 
children’s and adults’ folk models would talk about phenomena using this term but 
instead employ other terms such as power or force. 

Many scientific concepts have their origin in everyday language and indeed have 
developed out of everyday use. This is so for many concepts in the natural sciences 





? All flight examiners participating in the research program, training managers, and airline consultants 
used the term situational awareness. Consistent with the research literature, the term situation awareness 
is used in this book unless there are special reasons, which are apparent from the given context. 
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(e.g., energy from Gr. energeia, or velocity from Lat. ve/ocităs) and mathematics 
(e.g., cylinder, from Gr. kulindros, roller). In aviation, too, many concepts-in-use have 
their origin in mundane language, and even when there are scientific equivalents, their 
actual use tends to be much broader than that in the sciences. This is even more so the 
case when the concepts are from the social sciences, as these can be found in everyday 
sociological and psychological explanations of behavior. “Airmanship”, for example, 
has been dated to a New York Times article in July of 1859 (OED 2016), where it was 
used to denote the behavior manifested by the balloon pioneers J. LaMountain and 
J. Wise. It was not created as a psychological concept to name a particular phenomenon. 
Nevertheless, when pilots make a pan-pan call after discovering that the batteries have 
heated and do not cool down, flight examiners might explain this by saying that the 
pilots have good airmanship and that they are displaying the thinking that goes with 
the term. Flight examiners may describe small errors as “brain farts”, and, in so doing, 
manifest a particular folk theory about the nature of errors. When new tasks or new 
ways of operating are introduced into the recurrent training and assessment cycle, 
flight examiners might talk about the “learning curve” that the training department has 
arranged for every pilot involved. Or they may use the same term for describing what 
they and the entire training department is undergoing with the introduction of a new 
procedure. Thus, when flight examiners use terms such as “situation[al] awareness”, 
“management”, or “decision-making”, there are no guarantees that these uses correspond 
to how psychologists and cognitive scientists use these same terms. 

Folk models, even if they are branded to be and dressed up as scientific (e.g., by 
creating some visual), may have considerable problems from a scientific perspective. 
This is so even if practitioners find such models appealing (e.g., Ford and Munro 
2011). Suppose we are dealing with a seven-concept model organized into three levels 
with arrows somehow pointing into space as this often occurs in the representations of 
practitioners of all types (Figure 1.10). Consider the relation between any two concepts, 
concept, representing situation awareness and some lower-order concept,, which is 
supposed to capture the accuracy of flying an aircraft within the given tolerances. Let 
us call it the skill of “flying an aircraft within tolerances” (Mavin and Dall’ Alba 2010). 
A first problem lies in the concepts themselves. What do these concepts stand for? If 
two flight examiners look at the same situation but disagree whether or not it exhibits 
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Figure 1.10 The basic structure ofa model with seven elements and arrows somehow pointing into space. 
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poor situation awareness, then surely there is a problem from a traditional scientific 
perspective. In the performance and assessment model where the two concepts appear, 
both are presented as “skills”. However, in the psychological model of the foremost 
expert on the issue, situation awareness involves perception and comprehension of the 
current environment and the projection of the current on future states (Endsley 1994b). 
That is, situation awareness is coupled with the environment and, thus, by definition 
not a skill that can be located on the person pole of the pilot-environment relation. 
Similarly, limits and tolerances are specific to situations; and the specification of limits 
and tolerances are part of the rules in the cultural-historical activity theoretic approach 
not part of the subject (Figure 1.9). 

A second problem with this relation between concept, (e.g., situation awareness) 
and concept, lies in the arrow. What precisely does this arrow indicate and stand for 
when it does not connect two concepts (e.g., by a line)? Does it indeed point to the 
second concept in the sense of a determinate force that has a causal effect? Or does the 
arrow point merely downward in the same way that all the lower arrows generically 
point upward? Does concept, (e.g., situation awareness) affect concept, (e.g., flying 
of an aircraft within tolerances)? Even more importantly, if an aircraft approaches or 
transgresses given limits and tolerances then this becomes part of the flight examiners’ 
evolving objective sense that document issues with situation awareness, the content of 
their documentary sense (see Chapter 4 and Chapter 7). That this is so can be seen from 
the fact that situation awareness was the only human factor noted in the crash of Air 
Bagan flight W9011 (see box in Chapter 7), where the aircraft deviated horizontally and 
vertically from the flight path given on the approach plate and described in the airline’s 
standard operating procedure. If anything, Figure 1.10 expresses part of what in Chapter 
2 is described as the documentary method of interpretation whereby a culturally shared 
concept (e.g., situation awareness) is hypothesized to exist because of the observation; 
and, at the same time, it is used to explain that very same observation that has been 
used to postulate problems with situation in the first place. There is therefore some 
circularity at work that is not expressed in the folk model (Figure 1.10). 

In the Tasman region, a number of airlines use a model of assessment of pilots’ 
performance branded to have a scientific basis (Ford and Munro 2011; Mavin and 
Dall’Alba 2010). It includes situation awareness, flying within tolerances, decision- 
making, communication, knowledge, and management as the main concepts. As 
suggested above, the reanalysis of the original data shows, however, that the model is 
not scientific at all but merely represents the mundane ways in which flight examiners 
tend to talk about pilot performance (Mavin and Roth 2014a). Indeed, rather than 
using arrows that point into some space, the reanalysis shows how flight examiners 
use the concepts together to produce explanations of pilot performance. This can be 
seen partially in Scenario 1.5, where, on the one hand, the pilot flying is characterized 
as having lost situation awareness, but where the aircraft was extracted because the 
PM, aware of the situation and having good knowledge, “picks up the error”; this error 
detection is made possible because the PF nevertheless displayed good communication 
(i.e., verbalizes thinking). Here, it is actually communication that reestablishes situation 
awareness, a phenomenon that the original study on assessment does not allow for. 

Research on conceptions and conceptual change has shown that naive theories 
are rather coherent and consistent, with substantive predictive value and that folk 
theories are considerably robust, likely for that very reason (Roth 2008). But research 
also shows the existence of limitations of folk science when thought about in terms of 
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mechanistic theories. In this case, folk theories “are remarkably incomplete and often 
contain inconsistencies and contradictions” (Keil 2010). Moreover, common folk tend 
to be unaware of the limitations of their understandings. Simultaneously, in the formal 
sciences, gaps, inconsistencies, and gross simplification are observed even when 
scientists reason in their own domain but tend to be absent from their publications. 
Whereas there is much research on the different ways in which domain experts and 
novices or lay folk go about acting in particular problem contexts, there are both 
philosophical arguments and empirical studies supporting the idea that everyday and 
infant reasoning, as unscientific as it may appear, constitutes the very foundation of 
scientific practice. In the context of performance assessment, any attempt to get rid of 
folk models that may exist in current practice may actually amount to throwing out the 
baby with the bath water (Dekker and Hollnagel 2004). This is so because getting rid of 
folk assessment models may lead to the rejection of useful aspects that could be used in 
the construction of promising assessment models. Understanding the cognitive processes 
and cultural practices—e.g., discrimination, salience, decision-making—underlying 
folk models of performance assessment may be a first step in arriving at better models 
of performance and the assessment thereof. 
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KEY POINTS 


67 


+ Flight examiners use the documentary method of interpretation. In this method, an 


observation is treated as the manifestation of an underlying human factor that in 
itself is inaccessible (e.g., situation awareness, decision-making). A form of circular 
reasoning emerges because an observation, constituting the objective sense, leads 
to postulating the documentary sense, which, in turn, is taken as the source of the 
behavior observed. 

Over the course of a simulator session (sometimes extending into the ensuing 
debriefing meeting), the objective sense—as it pertains to each human factor 
considered in an airline’s (explicit or implicit) human factors model—evolves, 
accruing to a set of observations constitutive of the documentary sense. 

As the objective sense evolves, so does the documentary sense. Indeed, the 
evolving documentary sense (e.g., the pilot has trouble managing) shapes what 
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flight examiners are looking for. It even shapes the problems that they configure 
into the simulator task for the purpose of observing whether there is an underlying, 
deeper issue with the skills of crewmembers that would lead to the performances 
actually observed. 


+ The documentary senses of the performance factors are considered together to 
constitute an explanatory framework. In this explanatory framework, situation 
awareness is a master concept. Even when the concept shows up as a skill in an 
airline’s performance model, flight examiners treat it as a state. 


e With increasing know-how as flight examiner, the relationship between objective 
sense and documentary sense changes. Initially, the objective sense tends to be 
given more weight and frequently is tied to an airline’s assessment form; beginning 
flight examiners are oriented to individual facts. Later, the documentary sense 
tends to dominate and flight examiners are more oriented to understanding the 
“big picture”. 


+ Although flight examiners tend to focus on the assessment of individual pilots and 
may reject the idea of the cockpit as an irreducible, joint cognitive system, they 
tend to proceed as if they were using the joint cognitive system approach in their 
work. 


e Aptitude refers to the events at the pilot-environment interface rather than to 
knowledge and skills within the individual pilot. The methods of experienced flight 
examiners tend to be consistent with the way in which aptitude research operates 
to isolate the source of problems in complex systems. 


arly work on pilot assessment investigated assessment in terms of measurement 

models and focused on outcomes (e.g., Flin et al. 2003), where the measurement 

of performance sometimes was supported and enhanced by automated tools 
(e.g., Deaton et al. 2007). None of the identified studies investigate how flight 
examiners think, the method or methods that they use to arrive at statements about the 
proficiency, knowledge, skills, or cognitive states of the evaluated pilots. There exists 
considerable research in how experts from a variety of fields think, including clinical 
reasoning of medical experts, instructional designers, historians, and scientists. Such 
studies exhibit the tremendous importance of subject-matter specific knowledge in the 
forms of reasoning observed, knowledge that sometimes is not apparent because it is 
encapsulated in the practical know-how that experts have developed over the years. 
As shown below (Chapter 3), flight examiners and pilots at other ranks tend to vary 
in the facts they perceive and on which they base their assessment. First officers and 
captains, some of who eventually may become flight examiners themselves, are going 
about assessment in ways that differ from flight examiners: the former move item by 
item through an assessment form and identify performances that allow them to give a 
score on that item whereas the latter first construct narrative descriptions and then map 
the result onto the assessment form (Roth and Mavin 2015; Roth et al. 2014b). Both 
of these studies investigate pilots’ assessment of short scenarios videotaped in flight 
simulators while the pilots fly segments that are part of their company’s schedule. Such 
studies, though well designed, tell us little about how experts in assessment—flight 
examiners—actually think in the context of real situations where they observe pilots 
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over four-hour periods. This chapter is concerned with flight examiners’ methods of 
assessing pilots in real settings, such as the biannual recurrent training and assessment 
sessions that pilots have to undergo. The orienting questions include: “How do flight 
examiners think?” and “What evidence do they seek and use to arrive at an assessment 
and what are their thinking processes?” 

Here we are concerned with the methods flight examiners use. In their everyday 
assessment-related work, flight examiners do not and cannot return to the original 
sensuous praxis of flying that they earlier observed. The order of the world apparent in 
praxis is denoted by the term mimesis, (see Chapter 6). In describing, explaining, and 
assessing flight simulator events, however, flight examiners already have moved into 
the world of text that is ordering the observed world in particular ways based on their 
operating folk or scientific theory. Although this world of text constitutes a world of its 
own (its order is mimesis,, Chapter 6), it requires the users of the text to know their ways 
around the world generally. Practitioners and researchers alike take recordings—video, 
text, and graphics—as legitimate representations of sensuous praxis. It is precisely this 
move from the world of flying to the world of text that affords the latter to be a model of 
the former (i.e., practical action). Indeed, represented action itself appears in the form 
of text subject to all the requirements that have to be met for a text to be intelligible. At 
present, the concern lies with the interpretive method by means of which are derived 
the descriptions of pilot actions together with explanations in the form of human factors 
models of pilots’ performance—irrespective of the question whether these are based 
on scientific or folk models. 


THE DOCUMENTARY METHOD OF INTERPRETATION 


Much of the research on the work of flight examiners has been concerned with the 
accuracy or reliability with which they make assessments (e.g., Holt et al. 2002). Such 
research is concerned with the outcomes of assessment, the scores, and the comparison 
of these scores with the scores of other raters. The research is not concerned with the 
actual, sensuous work of doing the assessment. This poses the question: “How do flight 
examiners assess pilots?” Consider the following scenario that captures what flight 
examiners might do when they have an explicit human factors model that includes 
“communication”, “management”, “(aircraft and procedure) knowledge”, and “technical 
flying within stated limits” among its main factors. In the scenario, we take the place 
of flight examiners, who observe pilots in the simulator, where they have a viewpoint 
similar to that from which the pilots in Scenario 1.1 are seen. 


Scenario 2.1 


As I am taking notes in the flight simulator, I put beside the description a “C” for 
communication or an “M” for management. I make notes on both good and bad 
aspects of the flight: examples of good communication or management and how 
these affect the flight as well as examples of poor communication or management. 
Considering communication, if there are many examples where pilots mean one 
thing but they say something else, like when something is wrong, then I put these 
down to communication. The different technical areas—such as aircraft knowledge, 
standard operating procedures, or flying within limits—usually are quite easy to 
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judge, although what pilots say sometimes has to do with their ability to actually 
recall what they know. I then have to ask myself: “Does this belong to the category 
of situation awareness or to that of knowledge?” So we can summarize a flight 
examiner’s work like this: We are using specific examples to cover a generic fix. 


This scenario contains the essence of the method flight examiners use: For any 
one component of the assessment, such as communication, management, situation 
awareness, or knowledge, they make some “generic fix”. They then collect “specific 
examples” “to cover the generic fix”, that is, they use concrete observations as evidence 
for documenting and firming up the generic fix. As a result, there is a constitutive 
relation between concrete observations and presumed but unobservable (cultural) 
phenomena that collect a variety, often quite diverse observations together into a 
single category. This is the essence of what has been called the documentary method 
of interpretation (Mannheim 2004). In this method, observations are taken as evidence 
for, or documents of, an underlying reality while using this reality as a resource for 
explaining or interpreting the observation. The method corresponds to the mundane 
idealization of reality. All flight examiners, without exception, use the documentary 
method of interpretation whether they have or do not have an explicit, airline-wide 
performance and assessment model (Roth 2015b). Indeed, the documentary method 
is a mundane form of reasoning, which is not hidden in the mind but accessible in the 
concrete practices of flight examiners making or explaining performance assessment. 
Flight examiners are in the same situation as traffic court judges, who are confronted 
with two accounts of the real or actual traffic scene provided by a defendant and a police 
officer. From the perspective of judge, the real scene of the traffic violation is taken as 
independent of the two accounts, and yet it is available to the judge only through these 
self-same accounts (Pollner 1987). As a consequence, there exists an indeterminate 
relation between “what really happened” and all available accounts. In a way, this 
observation already anticipates the phenomena described in Chapter 3, where pilots 
assessing their peer not only use different human factors to reason about the same flight 
events but also differ in the facts they actually observe. 

The scenario also suggests that this method has little to do with measurement in 
the sense of taking some instrument with a scale that is then held to the phenomenon 
to be assessed, in the way that this may occur when flight examiners are asked to 
assess relevant categories—e.g., communication or management—along a scale from 
1 (unsatisfactory) to 5 (very good). Instead, assessment takes the form of categorization. 
Flight examiners therefore are in a situation that is not unlike that of doctors, who are 
confronted with symptoms (facts) and have to find an underlying illness; and, like in 
the aviation case, different medical practitioners will differ in their identification of 
symptoms and the ultimate diagnosis. Like other cultural phenomena in the social science 
and humanities that are not directly accessible and only manifest themselves in one or 
another way—e.g., “Impressionism”, “Modernity”, “Enlightenment”, or “Millennials” 
and “Generation Y”—cognitive, psychological, or human factors categories in aviation 
are bound up with the concrete instances that serve as their material documents. 

The constitution of the documentary method is based on three levels of sense that 
pervades categorization: objective, expressive, and documentary sense. 
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Objective Sense 


The objective sense ofa situation refers to what an observer actually sees. For example, 
flight examiners identify presumably indisputable facts: a pilot has (not) pushed the 
go-around button, the precise speed (e.g., 145 knots or white bug + 10) of an aircraft, 
or the current torque readings (e.g., TQ = 90%). During debriefing, flight examiners list 
sets of observations that constitute their objective sense, the facts used in the assessment 
of the pilots. Consider a statement such as “You were not a long way out, about 34 
DME [distance measuring equipment]. You called ATC [air traffic control], and you 
then made a PA [public address]. You returned to about 240 or 250 IAS [indicated air 
speed] until 10 DME, where your speed dropped to 235 knots”. In this list of facts, the 
flight examiner includes where the pilots made the decision to turn around (35 DME), 
that there was an exchange with air traffic control followed by a public address. The 
aircraft was flying at a rate between 240 and 250 knots until it was less than 10 nautical 
miles (DME) away from the airport, at which point the indicated airspeed was 235 
knots. Such lists do not in themselves constitute evaluations but are (a) used to abduce 
the intentions expressed and (b) treated as the concrete manifestations (documentary 
evidence) of one or more underlying, not-directly observable knowledge (aircraft or 
standard operating procedures), skill (manipulative, communicative, management, or 
decision-making skill), or state (e.g., situation awareness). 

Although more imprecise and fuzzy and not as easy to judge as the technical areas 
(see Scenario 2.1), the non-technical dimensions of performance also are described 
in terms of concretely observable evidence. For example, during debriefing, flight 
examiners may describe the response times of pilots both during observations in the 
flight simulator and in the debriefing meeting: “The pilot’s response to everything is 
very slow. It is slow even when you talk to him in the debriefing. There is always a 
considerable delay. But when the response is forthcoming, it generally is correct”. The 
objective sense thus is based on descriptions that are such that they can or could be 
verified, especially if the appropriate technology is available (e.g., a debriefing tool 
that records the entire simulator session). But, in this particular instance, slowness can 
be verified even outside the simulator, therefore providing additional documentary 
evidence to support the assessment that this pilot tends to be slow in thinking and 
execution. Every additional observation that this flight examiner makes consistent with 
slowness contributes to the constitution of the pilot as being too slow (“a bit dense”, 
“a plodder”, “a slowpoke”). Such documentary evidence on which the objective sense 
is based does not in itself determine the specific underlying but inaccessible category, 
for timeliness may appear in the context of different assessment categories, such as in 
the timeliness of communication or timeliness of decision-making. But in each case, 
it contributes to affirming the assessment that the pilot is (too) slow, though generally 
correct in the actions that are actually taken. 


Expressive Sense 


Flight examiners often make statements in which a concrete observation is taken as 
indicating an underlying intention that itself is not observable: The observed fact 
is an expression of this intention. This is quite apparent in the following statement: 
“You initially identified the propeller unscheduled feather. When you reviewed it after 
retracting the flap, you actually called it a prop over speed. This is not the message you 
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intended to give”. Here, there is series of observations—identification of unscheduled 
feather, retraction of flap, review, and statement “prop over speed”—all of which could 
have been verified in simulator facilities where a debriefing tool is available. These 
observations are followed by a statement about the intention of the pilot, which itself 
is unobservable. But the observations are taken as documents of the intention all the 
while the inference of the intention is based on those same observations. 

The assessment of what a pilot intended doing varies considerably when different 
flight examiners look at the same event. Thus, for example, while observing the video 
clip featuring the Missed Approach scenario (box, Chapter 1), the flight examiner pairs 
assessing pilot performance find manifestations for different underlying intentions on 
the part of the pilot flying (PF). In the scenario, the PF begins the final turn on a circling 
approach and the electronic attitude director shows the bank angle to be close to 30 
degrees (see Figure 1.5). When the IMC is called, the bank angle returns to 0 degrees 
while the pilot can be heard to say, “Missed approach was right hand, wasn’t it?” Hearing 
this statement, some flight examiner pairs note the pilot to be (a) actually turning to the 
right, (b) beginning a right turn but intending a discussion whether it actually it should 
be such a turn, (c) intending to go right, and (d) confirming the direction of the turn 
(Roth and Mavin 2015). Assessing pilot performance in another scenario where low 
fuel reserves are considered following a missed approach (see Chapter 3), the captain 
(PM) and first officer (PF) can be heard discussing whether to retry the approach or go to 
another airport. Among other things, the captain can be heard saying that he did not want 
to influence his counterpart, and the first officer acknowledges hearing what the captain 
is saying but expresses worries about the weather at the alternate landing site. Assessors 
watching this scenario all see the same event. But they take it as manifestations of the 
captain intending (a) an open discussion, (b) to listen, (c) trying to talk the first officer 
into compromising fuel reserves, (d) exerting subtle pressure, and (e) capitulating to 
the first officer (Roth et al. 2014b). In this case, there is objective evidence: Pilots in the 
scenario have an undeniable verbal exchange. There is no debate about the objectivity 
of the exchange. As discussed and theorized in Chapters 3 and 4, flight examiners differ 
in the “generic fix” that the documentary evidence is seen to be evidence of. Because 
any true intention is as unobservable as the constructs derived from the documentary 
sense discussed in the subsection that follows, we may treat them as aspects of the latter. 


Documentary Sense 


Over the course of their observations in the simulator sessions, flight examiners build 
up a holistic sense of pilots, the evaluation of their skills. Beginning with “a generic 
fix”, that is, some initial vague sense, they build up a documentary sense that explains 
the actual performance. For example, when in an icing condition during a single-engine 
approach, the pilots do not enter the correct speeds in their landing speed card, where 
all speeds (V rp Vi pp Vac» and V,,) should have been the same, flight examiners may 
attribute this observation to management skills generally and take it to be an expression 
of poor time management specifically. As a result of poor time management, pilots 
forget entering the speeds as per operating procedures. But these observations are used 
to establish the generic fix about management that is in turn said to be the case of the 
observed behavior. Flight examiners have no direct access to some management skill 
or to the long- and short-term memory of the pilots to check whether entering speeds 
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was actually stored there and whether there was a failure to bring the stored item into 
short-term memory (or working space). But the objectively observable evidence is taken 
as a document of what cannot be directly observed, the sense that documents itself and 
is documented in the evidence. 

In the work of flight examiners, three categories of idealizations are observable: 
non/proficiency (pass/fail), (non/technical) skills and knowledge (e.g., handling, 
decision-making, management, communication), and states/processes (e.g., situation 
awareness, thinking). All of these idealizations are mundane and uncontroversial cultural 
objects denoted by the language shared within the aviation community or specific airline 
(e.g., Mavin and Roth 2014a). These idealizations are integral to the folk models of 
pilot performance and constitute the common ground in conversations between flight 
examiners (see Chapter 3). These idealizations are taken to be the underlying root 
phenomena of the actually observed performances. The root phenomena are not observed 
but are derived based on a sense that arises with observations. Flight examiners may 
say that they have the “gut feeling” that something is not right or that something “does 
not sit with them”. Beginning with such a vague sense, flight examiners then seek 
specific observations that would confirm, and sometimes disconfirm, the presence of 
the phenomenon that appears to announce itself in the objective sense. 

The essence of the documentary sense, therefore, is a presupposed phenomenon 
taken to underlie what is observed. Conversely, what is observed is taken to be the 
manifestation of the underlying phenomenon. There is therefore a reflexive (circular) 
relationship between concrete observations and idealization of the underlying reality 
(phenomenon): the former lead to the emergence of the latter, but the latter explains the 
presence of the former. This mutually constitutive relation is quite apparent in debriefing 
talk, as exemplified in Scenario 2.2. 


Scenario 2.2 


In your decision-making, you considered all possible risks. I saw evidence of that 
early on in your decision to divert to Beantown when the hydraulics were gone and 
you had to do a flapless landing. You stated: “We need the help from engineering 
and we need a runway of sufficient length”. The runway at Leechtown is shorter than 
that. This eliminates Leechtown from your considerations. Beantown obviously is 
the closest airport; and it has sufficient runway length. In this situation, I see clear 
evidence that you were actually diagnosing the situation and were making sure that 
you considered all the facts required for generating all possible options. Acting in 
this way enabled you to make a correct decision. 


In this scenario, the flight examiner names the underlying category from the 
human factors repertoire: decision-making. It is positively evaluated, as apparent in the 
statement that the pilots had considered all risks. The flight examiner then formulates 
to have evidence for this assessment before providing a description of the documentary 
evidence in terms of the concrete actions and statements that the pilot has made in 
a particular event. The statement about the runway at Leechtown and the required 
assistance from engineering are taken as evidence for the underlying consideration 
and diagnostic skill, which in this situation is a subcategory of “decision-making”. 
The ultimate assessment of decision-making for an individual pilot is the result of the 
cumulative evidence of the different situations in which the underlying phenomenon 
manifests itself in some concrete form objectively available to any observer. 
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This phenomenon may also be expressed in terms of a cause-and-effect relation, 
which no longer is independent because the two terms are constitutive of each other: 
There is an effect (the aircraft followed the correct flight path at all times) because there 
is a cause (good situation awareness); but there is a cause (good situation awareness) 
only because there is an observable effect (the aircraft followed the correct flight path 
at all times). This relationship should not be taken as constituting a weakness in the 
approach and work of flight examiners, which some psychologists might ascribe to 
be an observation bias typical for non-academic practitioners. Instead, this relation is 
a condition for any categorization to be possible. In the science of logic, this form of 
reasoning is referred to as abduction (e.g., Eco 1986). In this form of reasoning, some 
unknown rule or law applied to a case leads to a specific result (observation). The 
process begins with a specific observation, such as the talk about alternative airports 
for doing a flapless landing, where the pros and cons of each airport were talked about. 
In a creative process, some rule, law, or frame of reference is generated as hypothesis. 
The rule, law, or frame is then applied to a specific case: the hydraulics failure requires 
a flapless landing. The rule here corresponds to good decision-making—which may 
correspond to a 4 (good) on a rating scale that goes from 1 to 5. Ultimately that rule 
explains the observation. 


EVOLUTION OF THE OBJECTIVE SENSE 


To arrive at conclusions about pilots’ proficiency levels, knowledge (non/technical) 
skills, or states (situation awareness, thinking), flight examiners require objective 
evidence on which to base their assessment. In the simulator, they observe and generally 
take some notes in real time; there is no time out or recourse to revisiting an event. These 
notes are used both for establishing the record of the flight performance observed and 
for the debriefing, where flight examiners often describe to the pilots what they (pilots) 
have done for the purpose of critique or praise. These observations are of the verbal type 
whether recorded on paper, by means of a debriefing tool (video), or in more-or-less 
verbal and initially more-or-less developed descriptions of the original event that now 
recedes into the inaccessible past. Any assessment thereby is based on the model of text. 

In their work, flight examiners differ in terms of which facts and how many facts 
they identify. The assessments are based on documentary evidence. One might ask 
whether flight examiners identify the same kind and number of facts. This is difficult 
to establish in the context of regular examinations but can more easily be done under 
controlled conditions when all flight examiners are shown and evaluate the same flight 
segment (see Chapter 3). An obvious drawback is the time requirement that is involved 
in observing a line-oriented flight training (LOFT) session, where crews (of regional 
airlines) may fly an entire segment and which may therefore last about an hour. Studies 
using videotaped scenarios tend to use shorter episodes, which has the obvious limitation 
of providing only some information so that the flight examiners feel handicapped. As 
shown in considerable detail in Chapter 3, even when flight examiners work in pairs, 
the number and type of facts identified and used varies a lot. In general, there is little 
debate about what the standard operating procedures say and whether a pilot action 
is consistent or inconsistent with these written documents. Such information is of 
factual nature with little variation in how it is interpreted. In the subsection “Expressive 
Sense”, it is noted that the flight examiners differ with respect to the interpretation of 
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the observable facts: bank angle and qualified statement about the turn. At the same 
time, only some flight examiner pairs watching this scenario actually noticed initially 
that the pilot flying had not pushed the go-around button. If this fact became part of 
the talk, it was based on subsequent observations such as the state of the flight director. 
This is consistent with the available information according to which the aircraft was on 
a positive rate of climb whereas the command bars directed the PF to head downward. 
It is not as if flight examiners are not aware of the fact that they attend to different 
facts and that they do so to different degrees. That is, they tend to be aware of the 
limitations of the factual evidence that they obtain. Flight examiners sometimes find out 
in the debriefing conversation that they have missed something or report themselves as 
having missed a particular fact (e.g., “I failed to note the point when the autopilot was 
turned off”). In part, situations in which the flight examiners do not identify important 
facts arise while they take notes and, therefore, have their heads down. That is, while 
writing notes they may miss one or more potentially relevant flight-related facts. As 
a result, there are situations when flight examiners and examined pilots differ in their 
assessment of what actually has happened. Flight examiners know about the inherent 
contradiction in their task: To fix the objective evidence that they need to support a pass/ 
fail decision or their assessment of underlying skill levels, they need to physically record 
their observations. But in the production of such notes, they miss out on observing flight 
relevant actions. Some flight examiners explicitly focus on observation while taking 
the scantest of notes (1—2 pages accruing to some 15 observations). They subsequently 
review their notes and what they remember in addition, pulling together all of the 
information to arrive at an overall assessment as well as at assessments of categories 
of performance, some of which may require special attention. Other flight examiners 
may take more extended notes (up to 5 pages for a 4-hour simulator session). In either 
case, the notes are used for the purpose of triggering episodic memory that allows 
flight examiners to remember what happened and those facts that they are using in the 
assessment (see Chapter 3). Getting an accurate overall picture is the most important 
dimension of assessment work for all flight examiners. This overall picture sometimes, 
and especially in situations of doubt, is expressed in the most general terms: “Would I be 
comfortable with having my family members fly in the aircraft with this/these pilot/s?” 
Their answer to this question determines whether the examined pilots pass or fail. 
The conflict created by the recording and observing actions is mitigated to some 
extent by a debriefing tool (if it is available). This tool records the entire simulator session 
and includes a videotape of pilots, shows what pilots view, and features representations 
of instruments and actuators (see Chapter 8, Figure 8.1). The debriefing tool allows 
flight examiners to place markers in the recording for easy recall and replay during the 
debriefing session. The process of going from observation to assessment is mirrored in 
the use of the debriefing tool. A flight examiner may mark more than 20 events during a 
four-hour simulator session but only replay four of these marked clips during debriefing. 
The total number of marked events provides flight examiners with a selection of events 
to work from. Near the end of the examination, as the overall picture emerges, flight 
examiners then select those that they deem most valuable in terms of pilot learning. 
Some simulator manufacturers make it possible for airlines to program a set of buttons 
such that can be used to code events in terms of different human factors. That is, if 
flight examiners have the sense that their current observation provides evidence for 
a problem with situation awareness, then press the button “S”. Flight examiners then 
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are afforded to specifically select the episode according to the human factor that they 
intend to discuss, here situation awareness. 

Some readers may be led to think that the flight examiner method leads to bias. 
But the fact is that they do engage in targeted evidence collection seeking to disconfirm 
as much as to confirm an initial sense (i.e., and initially emerging “generic fix”). 
For example, flight examiners, having accessed the pilots’ records pertaining to the 
preceding examination, seek to ascertain whether there is evidence for a problem to 
have disappeared or whether it continues to manifest itself. The records constitute a 
form of external memory where objective evidence is available on the basis of which 
conclusions can be made about a possible change to have occurred. If a pilot had 
exhibited insufficient levels of performance during an engine failure after takeoff 
during the preceding assessment cycle and the problems recurs during the current 
cycle, then flight examiners might have reasons to investigate whether there are deeper 
issues.’ Rather than being biased, they actively seek documentary evidence for deciding 
whether an observed performance problem is a random or systematic occurrence. On 
rare occasions, an examination session is organized to have another flight examiner 
provide an independent assessment. In such cases, the examiners use specific events to 
collect evidence on the particular issues that the preceding examination/s had identified. 
They then obtain the observation that makes the overall decision go one or the other 
way (i.e., pass or fail). 

To further confirm their sense with evidence, flight examiners may choose specific 
events from the simulator options such that the evidence required in support of their 
documentary sense is produced. They already know the human factors area that they 
want to evaluate and, at a difficult time during the flight, therefore introduce a specific 
malfunction that is thought to produce the observational facts required for determining 
the performance level in that specific human factor. In the course of a simulator session, 
therefore, flight examiners obtain multiple pieces of evidence to support their assessment 
of an underlying factor. For example, a pilot has kept the aircraft well within the required 
limits for the remainder of the examination, which should have been sufficient evidence 
for a 4 (good) rating on the technical skill of flying within limits. Yet flight examiners 
may tip toward a rating of 3 (satisfactory) because of a particular observation that the 
aircraft was at the lower limit during a non-directional beacon approach. That is, that 
one observation becomes decisive between a satisfactory and a good rating. In Chapter 
4, mathematical models are provided to explain the phenomenon of such flipping from 
one to another assessment category. 

On the job, flight examiners do not have the leisure to review some flight event 
repeatedly. But they do not identify arbitrary facts. Instead, in programming specific 
events into the simulation, such as an engine failure at takeoff or a sudden IMC requiring 
a go-around procedure, they ready themselves to observe specific facts that generally 
are associated with this type of event. Moreover, in a particular examination cycle, all 
pilots fly the same line-oriented flight segments and do the same spot checks. From 
delays in required actions, they anticipate workload to increase and pilots to come under 
time pressure, which may result in a loss of awareness of the situation as a whole. That 





' Depending on country and culture, there may be deviations from this pattern. Thus, for example, in 
the crash of TransAsia Airways Flight GE235 (February 4, 2015) that killed 43 occupants (Chapter 
10), one of the two captains at the helm (also PF) previously had failed examinations and has had 
recurrent issues with failed engines and with situation awareness (Aviation Safety Council 2016). 
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is, flight examiners’ attention is preconfigured by the choice and timing of events. This 
preconfiguration affords the anticipation of specific facts related to particular human 
factors areas that are more salient than others. Each event therefore is seen to provide 
a set of challenges (“boxes”), and the flight examiners observe whether they have 
ticked every box. 

The account provided so far may sound as if flight examiners do their work in 
an unprincipled and biased manner. But this is not inherently so. To make their cases 
for the presence of particular levels of proficiency, knowledge, skill, or state, flight 
examiners require objective evidence that can be taken as documents for the skills they 
have to evaluate and assess. They do not take single instances as cases for proficiency, 
knowledge, skill, or state. Flight examiners use repeat performances to increase the 
amount of evidence available as grounds for their reasoning. This is especially important 
to them in those cases where the observed performance requires possibly assigning 
a failing grade to the pilot. Moreover, flight examiners of multi-crew aircraft do not 
observe individual pilots but observe a joint cognitive system (see Chapter 1). Because 
observations of repeat performances are integral to the work of making attributions to 
individual components of an irreducible system, their use is elaborated in a subsequent 
section entitled “Joint Cognitive Systems and Assessment of Individual Pilots”. 


EVOLUTION OF THE DOCUMENTARY SENSE 


The movement from the observations flight examiners make to the categorizations 
implicit in the documentary sense may be understood in terms of the process of 
classification that brings into alignment fuzzy observation sentences and fuzzy categories 
(see Chapter 4). That constitutes a fundamental double bind in their work, which has 
also been described in the work of (academic and practicing) biologists attempting a 
classification that we may paraphrase for the present purposes in this way: in order to 
know what flight examiners are seeing, they have to know the appropriate classifier; 
but, in order to know which classifier is appropriate, they need to know what they are 
seeing (Roth 2005). Just as the scientists in that study “did manage to devise practical 
solutions to their classificatory problems”, flight examiners routinely manage to relate 
their observations and the airline’s human factors categories despite the ill-defined 
(fuzzy) nature of the latter. 

What flight examiners perceive is expressed in observation sentences, especially 
when they (have to) give a reason for this or that assessment (a) when they debrief 
pilots and explain what they did, why they did well, why they failed, and (b) when 
they ultimately write a report. Thus, when flight examiners say, “You were about 34 
DME from the airport”, they articulate an observation in the form of an observation 
sentence (statement). Such statements in themselves are not false qua statements; they 
may be intelligible in themselves and therefore may make sense (e.g., Schiitz 1932). 
But these sentences may not correspond to the objective situation at hand; that is, they 
may not have a referent (equivalent) in the world. The same object in the world may 
be described by means of many and even very different observation sentences, such 
as the well-known perceptual puzzles where the same pencil drawings can be seen in 
radically different ways: (a) a duck or as a rabbit, (b) an old woman (witch) or a young 
noble lady, (c) two faces or a vase, and (d) a cube protruding to the front or receding into 
the back (Necker cube). In most instances, once persons have seen some pattern, they 
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will not continue trying to find a different pattern. Whatever the first-seen pattern is, it 
will be the pattern (i.e., fact) for the purposes at hand. If multiple perceptual forms are 
possible in the case of simple drawings, we should anticipate the appearance of multiple 
forms in more complex situations involving psychological and sociological events. 

When multiple observation sentences come to be related by means of connectives— 
e.g., above, before, between, and after—observational predicates are obtained (Quine 
1995). When flight examiners say, “Your speed dropped”, they have connected and 
expressed multiple observations of speed into one predicate. 

Observation categoricals are statements that express something in the form of 
rules. Any observation categorical expresses a simple theory. Thus, in the course of 
observing pilots in simulator sessions, flight examiners make many observations, which 
may be expressed more or less explicitly in related observation sentences. Observation 
sentences that are related to one another in a consistent manner and thereby explain 
a form of behavior constitute an observation categorical. Such categoricals can be 
tested empirically—which flight examiners often do by creating specific conditions 
that allow them to confirm whether a specific behavior will be observed (see below). 
It has been noted above that the documentary method of interpretation is a common, 
everyday method for determining some assumed underlying pattern that also explains 
the observation—e.g., for finding out what someone thinks, for a coroner to determine 
the course of events that led to a death, or for sociologists to classify a set of data to 
find out how hospital records are kept (Garfinkel 1967). But precisely because it is an 
everyday method, it also is so powerful: the method not only helps in making sense 
but also intuitively makes sense. Flight examiners employ the documentary method 
of interpretation to determine whether pilots are non/proficient (pass/fail), what their 
non/technical knowledge and skills are, or the pilots’ situational awareness. All of these 
are phenomena are not given in themselves: these are cultural constructs that held to 
manifest themselves in observables. These constructs therefore exist only in and as 
documentary sense. 


From Vague Emergence of a Sense to Its Completion 


Viewing the same scenarios, flight examiners evolve different patterns taken to underlie 
performance given in documentary sense. This is consistent with studies that suggest 
considerable variation in the assessment of pilots when flight examiners are asked to 
assess the same videotaped scenarios (Flin et al. 2003; Mavin et al. 2013); and this is 
so even when pilots work in pairs (Chapter 3). That is, even in a condition where flight 
examiners discuss assessments in pairs—which should lead to diminishing the role of 
individual subjectivity—they arrive at very different conclusions about the performances 
and proficiencies of pilots. What are the reasons for such variations? 

Flight examiners do not have access to any human factor supposed to underlie 
performance or to pilots’ grasp and awareness of the situation (i.e., situation 
awareness). Here, as in the case of whether to pass or fail the pilot, flight examiners 
use the documentary method of interpretation. The human factors that flight examiners 
consider—aircraft knowledge, decision-making, management, situation awareness, 
and so on—are cultural objects, where the culture may be that of scientists or, more 
frequently, the commonsense world of pilots. Because cultural objects are not given 
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directly but indirectly through the manner in which they manifest themselves and 
because flight examiners differ in the contents of their observations that constitute the 
documents on which their objective sense is based (Chapter 3), the differences in flight 
examiners’ overall documentary sense become intelligible. The flight examiners are 
most concerned with overall proficiency. They might ascertain this overall proficiency 
at the most general level—e.g., by means of the above-noted response to the question 
whether they would want themselves or their family members and friends to be a 
passenger on the aircraft flown by that pilot or those pilots. If the response is yes, then 
the pilot passes; if no, the pilot fails. This is so independent of the root causes—i.e., 
human factors—attributed to non/proficiency. Thus, it is not surprising to hear flight 
examiners suggest that it does not matter which human factor is chosen as the root 
cause for the failure of a pilot as long as some (most appropriate) category from the 
assessment discourse is chosen. Even though flight examiners attempt to go about their 
work in a principled manner, there may be situations where their classification systems 
and decision-making are challenged, at which point they resort to relying on an intuitive 
sense of whether pilots under examination should or should not pass. It is that overall 
sense that also underpins flight examiners’ seemingly contradictory statements that they 
know pilots to “know the stuff” but that the latter fail to recall it. This is contradictory 
because failing to recall some fact means that the pilots have not produced it, which also 
means that there is no documentary evidence for that particular knowledge. However, 
because the overall sense concerning the performance or skills of a pilot evolve over 
the course of an assessment session, they also evolve a sense of what pilots know even 
when there is no direct evidence of this knowledge. 

The documentary sense begins with an indeterminate feel that unfolds, develops, 
and articulates itself over time into a more definite sense that is grounded in the available 
documentary evidence. The overall sense of whether a pilot is proficient or not, while 
evolving over the entire simulator session, may start as soon as the session begins. 
Flight examiners observe pilots over the course of a four-hour period and then make 
their assessments. That first, rather vague sense of how a pilot is doing emerges early. 
It may even start during the initial encounter in the briefing that precedes the simulator 
session; and sometimes that vague sense starts during the recurrent training session on the 
preceding day or days. Flight examiners know that the duration of the simulator session 
provides many opportunities for making observations that will constitute, develop, and 
ground their documentary sense and therefore may not feel hurried to make a decision 
early. In cases where pilots ultimately fail, there will be some point during the simulator 
session when a sense emerges that a fail rating is appropriate. Sometimes, however, 
the sense whether to pass or fail pilots has not become determinate even at the end of 
the simulator session, requiring further investigations into what pilots where thinking. 
A detailed descriptive and explanatory account of such cases is provided in Chapter 7. 

In the beginning, flight examiners first experience the emergence of a very general 
and generic sense without much concrete (objective) evidence description. This vague 
sense is apparent in descriptive expressions they may provide about a pilot who “is very 
introverted and either intimidated by the simulator or is intimidated by the process”, “is a 
plodder”, “is doing reasonably well”, “is getting through the task okay”. Sometimes flight 
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examiners may note that their sense begins with observations of pilots’ “body language”. 
Although sometimes used in the debriefing literature (e.g., Klair 2000; Tompkins 2000), 
the term itselfis controversial because body movements and positions do not constitute 
a definite code, do not have a grammar and, therefore, do not express conceptual content 
in the way language does (Nâth 1990).? Nevertheless, flight examiners may use the 
term when they make their initial fix. Then, as the examination session evolves, there is 
an increasing fixation and definition of the general sense, deriving from the increasing 
amount of concrete evidence available for use in the documentation of the ultimate 
assessment decision. Oftentimes flight examiners say with hindsight that the problem 
has shown up from the beginning—e.g., in the body language of the pilot. That is, 
what exactly they have observed changes through the lens of later observations and, 
more generally, through the lens of the evolving documentary sense. In this, how flight 
examiners see and assess some past event is not unlike how pilots in the debriefing room 
relate to what has happened (to them) in the preceding simulator session—as theorized 
in Chapter 6 and modeled in Figure 6.2. Thus, an earlier observation may contribute to 
a different objective sense when viewed through the lens of a documentary sense that 
has evolved over the course of the examination. If the documentary sense has changed 
from pass to fail, an observation initially not significant may now become significant in 
the light of the new documentary sense that constitutes a new overarching framework. 

The documentary sense integrates itself over: it is like a snowball. This is so because 
of its relation with the objective sense: a new observation is made through the lens of 
the existing documentary sense but in turn constitutes this sense either affirming or 
overturning it in the long run. In other words, the documentary sense is path dependent, 
its endpoint depending on its history (modeled by means of catastrophe theory, Chapter 
4). The evolving documentary sense shapes and is shaped by subsequent observations. 
When there is some event, it cannot be known whether the problematic performance 
will be recurrent. It is only with hindsight—after having observed numerous repeating 
events—that flight examiners will and do attribute the problem to some inherent 
shortcoming in the pilot, which leads to a fail rating. There is therefore a path dependence 
in the evolving overall sense concerning non/proficiency. In their words, flight examiners 
may describe the starting point of their sense in terms of pilots appearing flustered, 
at which point they frequently say something inappropriate. For example, pilots may 
categorize an event as “unscheduled feather” when it really is a “prop over speed”. This 
may then be seen—with hindsight—as a start to make mistakes eventually leading to 
a poor performance that has its start quite early in the session. 

Flight examiners seek further evidence (implicitly or explicitly) to confirm or 
disconfirm the current documentary sense. As Chapter 3 shows, in the attempt to locate 
specific facts, flight examiners tend to find more negative evidence. In none of the cases 
where pilots eventually failed or assigned minimum standard (poor) ratings is there an 
evolution from a more negative to a more positive documentary sense. Instead, flight 





? A useful distinction can be made between various classifications of gestures that differentiate 
movements in terms of a continuum ranging from idiosyncratic, spontaneous movements that 
accompany speech (gesticulations) to fully developed linguistic systems such as American Sign 
Language. The movements considered when people talk about “body language” tend to be idiosyncratic 
and spontaneous. They therefore do not follow any semantic or syntactic pattern and, as such, do not 
constitute a language. 
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examiners either begin with or move to a more negative sense concerning a pilot’s 
performance. That movement itself integrates itself over because past observations 
now are seen differently within the emerging horizon of the new sense fundamentally 
confirming it—which is simply an expression toward producing coherence based on all 
documentary evidence available. In the course of the simulator session (and, sometimes, 
extending over the course of debriefing as well), flight examiners ultimately produce 
an explanatory framework. 


The Production of an Explanatory Framework 


The work of flight examiners is part of a culturally and historically situated societal 
activity leading to specific outcomes (see Chapter 6); these outcomes, the products of 
the activity, become part of other activities within the extended system. At the end of 
the assessment activity, a record of the performance is produced. This record includes 
items from an explanatory framework. It is by means of the documentary sense that 
flight examiners evolve such an entire explanatory framework. When an airline has an 
explicit performance model, flight examiners evolve bit by bit a documentary sense 
for each of its components, situation awareness, management, and so on. Even when 
an explicit model is not available, situation awareness, management, decision-making, 
and handling may be among the features of the ultimate explanations of performance 
on which the pass/fail decisions are based. That is, an explanatory framework arises 
together with the general sense of overall proficiency that flight examiners evolve based 
on their observations. The documentary evidence and the associated objective sense 
configure the emergence of a documentary sense (e.g., level of situation awareness), 
which then explains the facts observed and is considered to be their cause. The associated 
cultural idealization (e.g., situation awareness) might then be explained by something 
else that is based on evidence (e.g., [workload] management). Thus, the fact that a pilot 
delayed some task might be taken as evidence that there are problems with management, 
which may have high workload as its consequence, which in turn lowers situation 
awareness. That is, taken together all the explanatory terms that flight examiners use 
in their assessment discourse—e.g., some subset of knowledge (facts, procedures), 
management, communication, decision-making, and situation awareness (Mavin and 
Roth 2014a)—constitute the explanatory framework based on the documentary method 
of interpretation. These explanatory frameworks, because of their intelligibility and the 
shared method used to derive them, are cultural objects through and through. 

The overarching explanatory framework, just as in the case of the documentary 
sense associated with individual categories, determines whether some observed 
fact is part of an underlying pattern or “momentary lapse”. That is, even though the 
observation of an error normally documents that there is an underlying problem “lapse”, 
the overarching framework determines how to classify the evidence. Thus, what might 
be a serious error in one instance—a turn onto a 15 nm arc when the required course 
asks for a turn onto a 10 nm arc or a right turn onto an arc when the aircraft should 
taken a left turn—may be (completely) disregarded if the preceding part of the session 
has gone well (e.g., see the case study in Chapter 8). Flight examiners might mobilize 
attenuating circumstances, such as the fact that the crew in question has had to hand- 
fly an entire session (i.e., without autopilot) because of some simulator malfunction. 
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In such situations, and given the absence of other problems, flight examiners may 
give a low but pass rating (e.g., situation awareness for a wrong turn)—even though 
under the conditions described in the Missed Approach scenario (Chapter 1), many 
flight examiners fail the pilot flying because of the expression of uncertainty about the 
direction of a turn. That is, on the job, flight examiners distinguish between momentary 
lapses and underlying problems, between actually observed performance (evidence) 
and the presumed underlying pattern. Flight examiners have to find out whether a poor 
performance is a manifestation of a fundamental problem or the result of something 
else. Consider a scenario where the pilot is moving forward in the direction of the power 
transfer unit switch, which sits right below the radio magnetic indicator, itself below the 
airspeed indicator. The movement, which can be objectively seen, is a manifestation of 
the intention to go for the power transfer unit (PTU) switch (see subsection “Expressive 
Sense” above). The movement stops as a radio call comes in, and the pilot attends to 
the call. In their everyday language, flight examiners might suggest that the pilot “has 
had a brain fart after takeoff and forgot to turn the PTU off because a radio call came 
through at that exact same time when his hand was going for the switch”. The pilot, 
upon returning from attending to the call, does not complete the action ascribed to 
the earlier movement. The sequence then becomes a manifestation of (documentary 
evidence for) poor management in this situation. How flight examiners might debrief 
such misses—which also are part of aviation incidents such the Ansett New Zealand 
crash involving a Dash 8 (Chapter 8)—is described and discussed in Chapter 8. 

In contrast to the way in which they make assessments under controlled conditions 
where brief videotaped scenarios constitute the object of evaluation, flight examiners are 
more flexible on the job where they observe pilots over two four-hour sessions spread 
over the two days of a recurrent training and assessment cycle. They tend to accept that 
pilots may have a bad day, account for the time when the observations are made (e.g., 
simulator sessions late at night or during the wee hours of the morning), and draw on 
whatever else is in favor of the pilots that may explain negative observations. This then 
allows understanding what might appear to be a curious fact about the predominant 
decision to fail a pilot based on a wrong turn under experimental conditions and the 
passing of pilots for making wrong turns under real, everyday conditions on the job 
(i.e., during recurrent training and assessment). 

In the course of producing an assessment, flight examiners employ the documentary 
method of interpretation even when an explicit assessment model and associated 
metric exists. The explanatory framework might be (a) the same as used in the airline’s 
assessment model, (b) other human factors-related concepts that characterize the culture- 
specific discourse of flight examiners (e.g., “automation management”, “manipulation”, 
or “compliance”), or (c) other concepts that provide an explanation for a range of 
concrete observations (e.g., “brain fart”, “airmanship”, “currencies”). Pass/fail decisions 
tend to be explained in terms of human factors-related concepts, themselves existing in 
the form of documentary sense. Flight examiners use verbal descriptions—“Ineffective 
organization of crew tasks” = 1 (very poor); “Controlled self or crew member actions, 
though with difficulties” = 2 (poor)—to translate between the performance seen and 
the rating scale. But they may change their ratings after tallying all of them. Thus, if 
the overall tally suggests a fail, flight examiners may still give a pass based on their 
overall sense. In this case, an individual score will be changed such that the overall 
tally ends up in positive territory. The reverse can also be observed. Thus, when a tally 
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suggests that a pilot should pass but the overall sense is that the pilot ought to fail then 
adjustments are made to individual ratings so that they tally is consistent with what has 
been determined based on the overall sense. 

Not all concepts flight examiners consider in the constitution of the overall sense 
are formal. Instead, they include informal cultural objects in the evolution of the 
overall sense of proficiency. Other, even more mundane explanations are observed. 
Thus, for example, flight examiners may relate all of their observations concerning 
handling techniques in different situations to the fact that a pilot also does the duties 
of a flight examiner and, therefore, is not or cannot be as current on technique as a 
regular line pilot. Lack of practice then is used as an explanation for some performance 
shortcoming. Flight examiners do know that there is uncertainty in establishing just 
what the observation is evidence of. The underlying skill may actually be present or of 
a particular level but is not expressed in that situation. 


Assessing the Missed Approach Pilots 


Situation awareness functions as a master concept in that it explains and is explained 
by all facts and human factors concepts that are documented by the facts. The 
following characterizes this bi-directional causation. Thus, for example, loss of 
situation awareness leads to wrong decision-making and wrong decision-making 
leads to “loss of situation awareness down the track”. The evaluation scenario is 
concerned with an unexpected Missed Approach on a circling approach even though 
the crew had earlier seen a raincloud in area where the final turn is to be made (box, 
Chapter 1). 


“The performance was not great in terms of the situation awareness displayed, 
when the flight management system (FMS) is in vertical navigation (V/NAV) 
mode and when the PF only is concerned with following V/NAV; and the PM 
has to prompt the PF to finish the configuration. The PF shows a bit of loss of 
situation awareness; he does not know where on the profile they are, and he is 
wrong about the altitude at eight DME. So attending to the flight management 
system and following V/NAV are wrong decisions, and there would be negative 
markers for it because it would lead to loss of situation awareness down the 
track. The major issue of concern in this scenario is the situation awareness of 
the PF. There is a delay in the circling call. Then they are on a left base into the 
airport. The torque is on thirty-four, thirty-five percent, which means that they 
could become unstable. The mountain is immediately to the right, the runway 
there to the left. The chart says that there is ano circling area to the right coming 
off the circling. During the initial part of the circling, the PF has the hands off 
the power levers and then loses situation awareness completely during the 
go-around, which is evidenced when he says that it is a right turn when there 
should have been a left turn, a turn toward the runway, as stated in the standard 
operating procedures. It is distracting that the PM has to say that it is a left turn. 
The PM is confused about the modes; and they seem to have forgotten about 
their flap setting. It goes downhill from there on. Normally people make one 
error, not multiple errors in the way this happens in this scenario. If we observe 
minimal standards, then a repeat is automatically required. So we have these 
pilots repeat the exercise”. 
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“Sometimes we debrief a performance like this immediately in the simulator 
and then redo it. Normally we wait until the end, which allows us to see the 
performance over the remainder of the session. The way in which the pilots 
flew the VOR, with nothing else aggravating the assessment, we would have 
bit of a word to them, debrief the approach, and then give them another go at 
it. We will then repeat the problematic approach, here the very high frequency 
omnidirectional range with distance measuring equipment (VOR/DME). Some 
of the things pilots are told to attend to include heading selects and altitude 
calls, and to make sure that there is a response to the “one hundred above” 
call. They also need to attend to the visual circling calls. We would query them 
to find out their recall and understanding of the go-around procedures. If the 
responses are satisfactory, we continue in the simulator, flying the VOR again. 
But if they still do not understand the procedures for the missed approach, how 
they differ when done from below or above circling, then we would leave the 
simulator being rated “total fail”. But if they understand their errors, the pilots 
will get another go at the VOR. Based on the single incident, they would not 
have failed the entire examination; they would fail only if there had been some 
trend, other things that added up”. 











Situation Awareness as Master Concept 


Situation awareness turns out to be a powerful master concept that is employed as 
part of explaining many aviation events (see box). Indeed, the term is often used as a 
shorthand expression to mark situations where flight examiners might say to pilots, 
“Make sure you remain attentive to the sounds in the cockpit!” or “Make sure to remain 
attentive to the light signals”. This is apparent in the following scenario, which exhibits 
crewmembers in a facilitated debriefing talk about an instance during the flight where 
there is a chime. 


Scenario 2.3 


01 PF : heard the chime at a thousand feet but you didn’t respond to it. 

02 PM: Ididn’t even know what it was until it started, and then I was like, “What 
was that?” 

03 FE : So be careful of your situational awareness that the chime came up. 


Here, the flight examiner might indeed be speaking about awareness, which 
justifies the adjectival use “situational”. But the flight examiner is indeed also making an 
assessment corresponding to the concept of situation awareness, which is derived from 
the observation that the chime had not been heard in the simulator; and, simultaneously, 
situation awareness explains why the chime was not heard. Ultimately, in the aviation 
community the documentary method is taken to amount to good story telling. Indeed, 
some flight examiners explicitly describe their method in terms of story telling. A good 
story binds together a large number of observations into a simple story line. In this, a 
good story is similar to a parsimonious scientific theory: it is convenient and convincing. 
The documentary method is used to create a big picture by putting together the right 
and required pieces of evidence that both create, hold together, and make plausible 
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the overall narrative; and it is the overall narrative that drives the selection of the 
individual pieces from the documentary evidence (objective sense). In the production 
of the assessment stories, situation awareness turns out to be a master concept (see grey 
box). It is not surprising then that situation awareness tends to show up in more formal 
and published incident reports, including those of the crashes of an Air Bagan Fokker 
F100 near Heho (Myanmar) (Chapter 7), a TransAsia ATR 72-600 in Taipei (Taiwan), 
or an Ansett DHC-8-102 near North Palmerston (New Zealand). 

In the scholarly literature, there is a debate whether situation awareness really exists 
or whether it is part of a folk model of human factors (see Chapters 1 and 10). Indeed, 
one may observe occasional confusion between scientific and folk models. Thus, the 
Model for Assessing Pilots’ Performance (MAPP) is presented as a scientific model 
integrating technical (“aviation knowledge” and “aircraft flown within tolerances”) and 
non-technical skills (“management”, “situational awareness”, “decisions considerate 
of risk”, and “communication”) (Mavin and Dall’ Alba 2010). As shown in Chapter 1, 
this “model” is not a model in the scientific sense, which requires connections between 
parts to be specific. As pointed out, this model actually represents a folk theory, a 
theory of the ways in which flight examiners talk about pilot performance. That is, the 
model constitutes a “grounded theory”, which is a concise summary of what research 
participants say rather than a theory that scientists evolve based on their definition of 
concepts and categories. It therefore comes as little surprise that the member checking 
procedure showed flight examiners to agree that they indeed used the model expressed 
in the grounded theory.’ Whereas the model does indeed feature situation awareness as 
something like a master concept, it fails in two ways: in that it (a) determines situation 
awareness to be a skill and (b) fails to account for the determination of situation 
awareness by factors including pre-conceptions, objectives, ability, experience, training, 
and so on (e.g., Endsley 1994b). The reinterpretation of the original data on which 
the MAPP is based indeed allows for the effects of other human factors on situation 
awareness (see Chapter 1). 

Flight examiners do not at all treat situation awareness as a skill in the way that 
the MAPP suggests. Instead, it is treated as a state that both affects performance and is 
affected by it. In the human factors discourse of flight examiners, situation awareness 
of pilots both explains observed performance and is derived from it. Flight examiners 
tend to be aware of the circularity in this method: If pilots do not have situation 
awareness, they would not be able to fly correctly; and flying correctly inherently 
means having situation awareness. Flight examiners might express this in their own 
words in this way: If pilots did not have good situation awareness, they would not be 
able to complete a specific task without error. Because one cannot measure situation 
awareness as such, the quality of situation awareness is judged based on the results 
of their situation awareness. Situation awareness thereby comes to get the status of 
a master concept that cannot be falsified. That flight examiners find it hard to assess 
situation awareness really does not come as a surprise: because it is a state, it changes 
with the situation and is affected by other things pilots do and by the way in which they 
perceive the current situation, including the outcomes of their own actions (Endsley 





> In other words, the original work had summarized what flight examiners say about performance 
assessment and then checked whether the same and other flight examiners agree with the summary. 
The resulting model therefore is not at all scientific but a summary of the common ways in which 
flight examiners talk assessment. 
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1994b). Flight examiners end up explaining this state in terms of other human factors, 
inaccessible directly but manifesting itself in concrete actions and performances. Thus, 
flight examiners might state that pilots could work on bettering their aircraft knowledge 
(technical) because that would enable their management (non-technical) and improve 
their situation awareness (non-technical). In Chapter 10, an extended teaching | learning 
episode is used to show what might be done in debriefing when flight examiners have 
flagged an event as featuring a lowered situation awareness. 


THE ROLE OF EXPERIENCE 


As flight examiners accumulate assessment sessions, the relationship between objective 
sense and documentary sense changes. This is apparent in the different ways in which 
beginning and experienced flight examiners go about their work and the different role 
the two forms of sense have in determining the assessments they make. Experienced 
flight examiners let the documentary sense dominate the objective sense. The overall 
assessment may override any other assessment, for example, derived by rigorously 
implementing an assessment metric. Flight examiners admit that they sometimes skew 
the results if they think a pilot deserves to pass but does not do so based on the ratings 
(see above). Here, minor variations in marking determine whether a particular dimension 
of the assessment model is a minimum standard (poor) or satisfactory, which entails 
a fail or a pass. There is an awareness of the subjectivity (indeterminacy, uncertainty) 
involved and that what is a 2 (poor, minimum standard) rating for one examiner might 
be a 3 (satisfactory) rating for another (see Chapter 4 on the effects of such minor 
variations). This is why the overriding sense governs whether a borderline case will 
pass or fail, and the documentary evidence will be adjusted accordingly. 

Less senior flight examiners tend to focus on individual facts. The debriefing 
sessions show that less experienced flight examiners tend to focus on what more 
experienced examiners refer to as “minutiae of small errors” rather than on the big picture 
(sense). When they have an explicit assessment model, less senior flight examiners may 
entirely rely on the assessment form and its metric. This is consistent with the observation 
that the assessments captains and first officers make are driven by the structure of 
the assessment tool whereas flight examiners are driven by their overall sense 
(see Chapter 3). Thus, beginning flight examiners, who tend to be more concerned with 
individual facts, find themselves assisted in looking for the objective evidence required 
for making a particular assessment. They may make extensive use of the behavioral 
descriptions to translate between observations (e.g., “Manipulated accurately, with no 
deviations from target parameters”) and associated scores (e.g., 5, very good). 

Over the course of a flight examiner career, the boundary between pass and fail 
performances becomes clearer. There are suggestions that sensitivity between adjacent 
scores is low in pilot examinations (Holt et al. 2002). The decision whether to pass or 
fail a pilot based on the performance during a simulator session, though never taken 
lightly, becomes easier with increasing experience. This is an analogue case to natural 
scientists’ increasing competence in classifying initially poorly distinguished natural 
objects; and this competence comes with an increasing number of cases that they 
classified in a study or over their career (Roth 2005). Flight examiners express that 
over time in the job they become more able to discern whether they are going to pass 
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pilots, which may occur even when they (momentarily) see something that is really 
poor. Looking back at their development, flight examiners realize that they passed or 
failed a pilot in the past that they would rate differently now. Although their overall 
sense between what constitutes the difference between pass and fail ratings sharpens, 
the differentiation between two adjacent scores remains difficult: Assessing pilots is 
not having two boxes that clearly divide a fail from a pass or a 2 (poor or minimum 
standard) from a 3 rating (satisfactory or acceptable standard). The situation is similar 
to the difference between a cup and a mug. There will be cases where it is not so clear 
whether the object still is a cup or already a mug, and vice versa. How to model such 
situations, where minor variations lead from one to another, qualitatively different 
behavioral state is shown in Chapter 4. 


JOINT COGNITIVE SYSTEMS, APTITUDE, AND PERFORMANCE 
ASSESSMENT 


In the course of the fieldwork for this book, flight examiners and consultants repeatedly 
pointed out that the joint cognitive systems approach (Chapter 1) does not make sense 
and does not correspond to the reality of the aviation industry where pilots are and 
have to be assessed individually. Observations in the field reveal, however, that flight 
examiner methods and practices do take into account the joint cognitive system as a 
whole. Flight examiners tend to vary parts of the system systematically for the purpose 
of observing the specific effect these variations have on other parts of the system. The 
following begins with the articulation of the psychological concept of aptitude that is 
highly appropriate in the context of considering assessment of performance of complex 
system in which humans are constitutive parts. 


The Concept of Aptitude and Pilot-Situation Transactions 


There exists a considerable amount of research generally categorized under the 
term “situated cognition” that exhibits the significant performance differences when 
situational characteristics are changed. One of the classic studies investigates the 
mathematics used during grocery shopping (Lave 1988). This research shows that 
performance drops from 99% accuracy on best-buy problems when participants are 
actually in the aisles of the supermarket to about 75% when the same shoppers are 
presented with items they considered buying placed on a table in the parking lot of the 
supermarket. The same individuals drop to about 50% correct solutions when the best- 
buy problems are presented in the form of a pencil-and-paper tasks. Such studies have 
led to the conclusion that cognitive skills are not constant but seemingly fluctuate as a 
function of the situations in which they are deployed. The immediate upshot of such 
findings is that skills, such as those presupposed in research on human factors, are limited 
in their generality (Roth and Jornet 2013). Such research also shows how thinking is 
intricately interwoven with the context of the problem to be solved; thinking involves 
the physical context, purpose of the activity, and the social relations within which the 
problem is situated. Thinking skills, based on such research results, have to be thought 
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as situated in persons-acting-in-situation transactions rather than in the mind modeled by 
means of the metaphor of a traditional digital computer. Moreover, rather than treating 
pilots and their environment as independent variables and variable complexes, the pilot- 
acting-in-environment needs to be taken as a irreducible, single and dynamical unit in 
which the different parts constitute one another (see Chapter | on transactional models 
of pilot cognition and performance). In such dynamical systems, for example, recalling 
a memory item to be completed following an engine fire after takeoff is not modeled in 
terms of a representation stored in long-term memory. Instead, recalling such an item 
is modeled in terms of a dynamical system settling into one of its attractors (e.g., van 
Gelder and Port 1995); and failing to recall means that the system as a whole does not 
settle at the relevant attractor. The mathematical models of such dynamical systems 
cannot be decomposed into separate components, individual and environment because 
the characteristics of one part also are characteristics of another part. 

Even before the research on situated cognition flourished, psychologists 
recognized performance and learning differences when the same individuals were 
studied in multiple settings. This has led to the articulation of the aptitude concept. 
This concept is understood as, and denotes, the irreducible unity/identity of person and 
environment (situation). Aptitude understood in this transactional manner addresses the 
observation that the effects of individual pilot characteristics on performance cannot be 
understood unless the performance situation is determined. Conversely, the effects of 
the environment in which pilots perform cannot be understood unless the characteristics 
of the individual pilot are specified. Paraphrasing an educational psychologist who 
specialized in aptitude treatment interactions (Snow 1992), we may say that aptitude 
differences in the learning and development of individual pilots appear at the pilot- 
simulator task interface as differences in within-pilot adaptations to the stream of 
continuing changes in within-simulator task demands and opportunities. Information 
processing models (see Chapter 1) cannot account for the complex transactions that 
take place at this interface of person and setting—in particular because as soon as 
some variable has been defined and measured, psychologists no longer are dealing with 
the system as a whole but with a single manifestation thereof. Manifestations change 
and the same dynamic system may manifest itself in qualitatively different ways (see 
section on activity systems). 

When pilots are considered to be integral parts of a transactional cognitive system, 
then any performance situation in the simulator can be understood as sampling from an 
individual pilot, who forms an irreducible, single unit with the environment that includes 
the other pilot and the manufactured part of the system. At the same time, the pilot 
under evaluation has to be understood as sampling the situation—not in the objective 
manner that has been formulated in the original ecological approach to psychological 
theory (Gibson 1986) but from the perspective of the environment as individual horizon 
(Vygotskij 2001). In this approach, the environment may provide the resource for 
the system as a whole to respond in an appropriate way (see Chapter 1). Thus, in the 
Missed Approach scenario (Chapter 1), the standard operating procedure calls for the 
pilot monitoring (PM) to select and call heading, lo bank, and IAS [indicated air speed] 
to VGA [go-around target airspeed] (see Table 1.1). But in the scenario, the following 
exchange can be heard. 
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Scenario 2.4 


01 PM: Gear up. Power is set and turning left, established on the two one zero. 
02 PF : Okay, flying on heading and IAS to VGA. 
03 PM: Heading IAS to VGA on. 


In this situation, we observe what practitioners refer to as one pilot “picking up” 
for another (cf. Table 1.1). In the system as a whole, the call and selection of IAS to 
VGA is completed but not on the part of the PM, as intended by the standard operation 
procedures. The system as a whole, PM-acting-in-environment performs, but the 
actions are shifted to the environment pole. This performance of the system as a whole 
is no different from that of using speed bugs, which constitute a form of memory that 
neither pilot needs to perform because it exists inside the system but outside of the 
pilots’ heads (Chapter 1). It would be too quick to attribute some failure or shortcoming 
to the PM. As soon as we consider the history of the flight, we observe that an initial 
disturbance emerges when the PF returns the aircraft to a bank angle of 0 degrees and 
asks for (seeks to confirm) the turn direction to be flown during this go-around. Thus, 
flight examiners watching the scenario consider the PM to be under pressure because 
of the disturbance brought about by the uncertainty concerning the turn direction; the 
source of the pressure is his environment. That is, the failure to call and set “IAS to 
VGA” is itself treated as a response to the situation that becomes input to the system 
as a whole, which then responds by producing the call in a different part of itself. 
That is, here “the situation contributes components to the performance that the person 
then need not provide” (Snow 1992). Everyday performance always functions in this 
manner, which is why even the most powerful computers have not come close to the 
special capabilities of humans. 

In the aptitude approach, disharmonies in the pilot-environment system (unit) 
are of equal importance because it is here that failure arises. Inherent in the object/ 
motive of recurrent training and assessment is the concern for system failure, and 
airlines in general and flight examiners in particular have as their goal the prevention 
of pilot-environment combinations that are more error prone than desirable. Simply 
testing a pilot in some abstract way using whatever test of general ability that may 
be considered appropriate—like the intelligence tests used to determine scholastic 
ability—is unhelpful. This is so because such tests, having been used and standardized 
on tenth of thousands and even millions of individuals, provide little information about 
the individual. Instead, using the same test with thousands or millions of individuals 
provides research with a lot of information about the test itself (e.g., consider how 
responses by many people are used to identify the structure of an instrument by means 
of factor analysis). To know a lot about an individual person—an individual pilot in 
the present context—requires observing the same individual across many different 
performance situations. Each performance situation is understood to be sampling from 
the individual pilot “in the sense that the demands and opportunities it presents draw 
forth whatever relevant response components and assemblies of components each person 
can muster” (Snow 1992). This leads to a collection of situation-specific performance 
descriptions. As shown in the next subsection, flight examiners’ methods already 
embody such an understanding of performance but, for administrative purposes, this 
method collapses all the information into a single score that is thought to provide the 
gist of the characteristics of the pilot considered as an element detachable from what 
is known as an irreducible joint cognitive system. 


64 Cognition, Assessment and Debriefing in Aviation 


Flight Examiner Sampling Methods 


Flight examiners know from experience that they do not assess pilots completely out 
of context. How pilots “look” depends on the events to which they are exposed and 
on the other crewmember. Thus, flight examiners will sometimes say that particular 
pilots looked good because of their partners, in any case better than they might have 
in other situations and especially better than with other (“normal”) pilots. They might 
say, while talking about a first officer, that the aircraft could have found itself in a very 
difficult situation, which would have overwhelmed the crew as a whole had the other 
pilot been a very average captain. But because the captain was very good, and provided 
outstanding supporting calls by interjecting early enough to counteract the otherwise 
slow responses on the part of the first officer, the overall performance looked better 
than it would have otherwise. In this recognition, the flight examiners intuitively draw 
on a transactional model (see Chapter 1), which recognizes that an improved overall 
performance also makes individual agents look better. 

Flight examiners do their work as if they were using the joint cognitive systems 
approach, even though they are tasked with and oriented to the performance assessment 
of individual pilots. That is, they act as if they were working with the aptitude concept 
and as if they were attempting to assess it. Indeed, they might contradictorily state 
that each pilot has a set of skills that then add up when they work in pairs in two- 
pilot cockpits. If the world were to work in this manner, then some other form of 
performance assessment could be chosen, possibly one much less expensive in terms 
of personnel- and facility-related financial costs. We can conceptualize what flight 
examiners when assessing a system as consisting of a set of analytically distinguishable 
parts. (Analytically distinguishable means that the parts are not in fact separable and 
separated elements ofa whole.) This set may be written as {P4}, Pr AG, AS, E, ), where 
P" and PR are the left- and right-seated pilots, 4% is the aircraft cognitive system, AS are 
physical states of the aircraft, and £ are the states of the environment. This set describes 
all those parts of the system that contribute to the events at the interface of humans and 
of humans and the physical environment. Assume flight examiners are interested in the 
performance of the right-seated pilot. They then have the option to observe P*-related 
performances across the same and different configurations of the remaining parameters 
by varying one or more of these. 

First, flight examiners conduct repeat observation for the same set of people, system, 
aircraft, and environment configuration. When performances are problematic potentially 
pointing to underlying issues, flight examiners ask for repeating a situation, segment, or 
exercise to collect further documentary evidence that allows them to get “a better fix” 
on a factor of interest (see the box in this chapter). Repeats provide further information 
about the underlying proficiency, knowledge, skill or state thought to underlie the pilot’s 
performance. If pilots perform sufficiently well during the repeated exercise, then this 
constitutes evidence that there was an issue with the particular performance (a “one 
off”) rather than with the underlying dimension. If flight examiners observe that the 
same mistake/s is/are made given the same exercise, then the problem is attributed to 
the underlying skill of that pilot who appears to have been at the origin of the problem. 
In so doing, flight examiners produce multiple forms of documentary evidence for the 
fact that an underlying ability is present but occluded in the performance. 
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Second, in the course of a simulator session, flight examiners observe the system 
in different environments while all the other parts of the system may be constant (i.e., 
E „ is manipulated). Flight examiners may then observe multiple situations in which 
an error appears to have its source in the pilot under consideration or the other pilot 
steps up managing some aspect. If they observe management issues across two or more 
exercises then the sense may emerge and be substantiated that these can be attributed to 
the same pilot. Or flight examiners may observe some recurrent behavior, such as a PF 
leveling out repeatedly without pulling the power levers back. They then have two or 
more observations of the same type but across different situations. These observations 
are then taken as manifestations of the same performance problem attributable to the 
individual pilot. Because the performance problem occurs across situations it is taken 
as a manifestation of an underlying but in itself unobservable skill. That concern for 
manipulative ability is more serious than the concern with assertiveness, for which the 
flight examiner may have had evidence that it can be fixed. 

Third, flight examiners configure specific failures in the different parts of the 
physical aircraft—e.g., an engine fire after takeoff, an aileron jam—or in one or more 
instruments, that is, they go through different AS. Whereas these are failures that flight 
examiners routinely program into the simulation for pilots to deal with, a sense may 
emerge that a certain PF is depending too much on the automated pilot. They may then 
fail the automated pilot to observe the response of the system when the aircraft has 
to be flown by hand. Indeed, they may turn the automated pilot on and off during the 
same LOFT, observing the pilot flying the aircraft in the two modes. The pilot may 
have to fly an ILS approach without the aid of the automation. Over the course of their 
observations, flight examiners create multiple opportunities to assess changes in the 
system that are attributable, from their perspective, to the pilot flying who may not 
have attended to flying the aircraft as the primary task but also engaged in other tasks 
that placed attention elsewhere in the cockpit. A conclusion therefore may be that the 
pilot under observation relies too much on automation. The recommendation will then 
be made for the concerned pilot to have opportunities at hand flying in the simulator 
or during line flights under the supervision of a training captain. 

Fourth, flight examiners may access the files of a pilot prior to the examination 
and find out that there had been some issue in the preceding assessment cycle. In all 
likelihood, the second pilot will be different. But flight examiners may configure the 
aircraft and environmental parameters in the same way so that they can observe the 
system generally and the target pilot specifically in what is taken to be the same situation. 
If the event unfolds without any significant issues, then the pilot is considered to have 
learned and improved. Otherwise, some deeper underlying issue may be hypothesized. 
Part of the work in the current assessment cycle is to better specify the source of this 
problem. If problems in the joint cognitive system with that pilot present persist over 
several cycles, involving different accompanying pilots, then flight examiners and 
training managers tend to come to the conclusion that there are deeper issues and that 
some skills are lacking. Moreover, if the same observations are made despite additional 
simulator training and flying with training pilots, then questions may be asked about 
the future of the pilot within the airline. 

Finally, flight examiners observe the system while “configuring” one of the two 
pilots by means of a tap on the shoulder, by asking, after having blocked out the pilot 
under investigation, the second pilot to provide only normal support calls or to fake 
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an incapacitation (i.e., they configure the second P). If the pilot under consideration 
performs to satisfaction then the reason for the performance is attributed to that pilot, 
who may be said to have stepped up when not having to deal with the other pilot. Or 
flight examiners may ask the second pilot—in a situation such as the NDB approach 
to Heho where the aircraft is off course on the outbound track (Chapter 7)—to let the 
first pilot work out the inbound track on his/her own but without telling this pilot what 
is happening. In this course of events, the possibility emerges for attributing the failure 
to properly track the approach not to the cockpit as a whole or to the two pilots but 
solely to the PF. The behavior of the PF then is taken to accounts for the deviations 
from the normal course, which presents flight examiners with the objective evidence 
that firms up the documentary sense and can be used in the documentation that will 
enter this pilot’s file. 

The work of flight examiners tends to focus on the assessment of pilots. However, 
this does not have to be the only type of work they conduct. The environment part 
of the aptitude may also become the focus of the investigation. That is, rather than 
varying parameters in the {P*, Pf, A (S, Aj, E, y set while observing one or both pilots, 
the motive of the observations may the role of the aircraft cognitive system. One such 
case has been reported in an airline’s investigation of any possible shortcomings that 
might derive from mixed fleet flying (Mavin et al. 2015). The term mixed-fleet flying is 
defined as the practice of “concurrent operation of different manufacturer, type, variant, 
or modification” (IFALPA 2015). In the study considered, the airline investigated the 
concurrent operation of two variants, a non-glass and a glass cockpit of the same type 
(manufacturer). A study was conducted where a crew with considerable experience on 
the non-glass variant has been trained and has accumulated several hundreds of flight 
hours on the glass variant. The crew was then asked to fly a non-glass variant in the 
simulator doing, over a two-day period, several targeted exercises and LOFTs. Pilots 
and flight examiner debriefed what has happened in the flight simulator in the presence 
of two researchers. 

The debriefing and the subsequent analyses revealed many errors not normally 
observed in the performance of these pilots specifically and pilots more generally. 
The study of these errors from a joint cognitive system perspective sought patterns 
in the errors by considering events falling into one of four cells when functionality 
(same, different) is crossed with physical layout (same, different) (Soo et al. 2016). 
Unsurprisingly, no errors are observed where both physical layout and functionality of 
an aspect in non-glass and glass cockpit are the same (e.g., flap levers). However, errors 
creep into performance in cases where even minor changes in the layout occur—such 
as minor, five-centimeter shifts of buttons and changes in the layout of buttons on the 
automatic flight control system and repositioning of the nose wheel dial—whereas 
the functionality remains the same. Other errors arise when operations in the standard 
operating procedure are assigned to a pilot (e.g., PM) but are completed by the automated 
system in the alternate version. This changes the coordination of the flows (kinetic 
melodies) of the different parts of the joint cognitive system, leading to what musicians 
call “being out of tune”. The 2 (same, different) x 2 (same, different) functionality x 
physical layout heuristic model, though used after the fact for the classification of all 
flight events that have been observed over the study period, constitutes an extension 
of flight examiners’ method to the study of performance aspects other than the pilots. 
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THE DOCUMENTARY METHOD AND THE ASSESSMENT OF PILOT 
PERFORMANCE 


This chapter exhibits the mundane ways in which flight examiners get their work 
done: by means of the documentary method. This method includes a translation of the 
sensuous world into a text (observation) that provides a foundation for their objective 
sense. A documentary sense, which explains the observed facts, evolves together 
with the objective sense. Flight examiners’ inquiries concerning pilot proficiency 
are based on the tight, reflexive relation between observational facts (evidence) and 
the mundane idealizations (cultural objects) that explain those facts from which the 
idealizations are derived. The chapter constitutes an answer to the question about how 
flight examiners reason, including during line-oriented flight examination (LOFE), 
operational competency assessment (OCA), or air transport pilot license (ATPL). They 
reason according to a documentary method of interpretation to arrive at a sense of the 
different cultural phenomena of interest—non/proficiency (non/technical), knowledge 
and skill, or state (situation awareness, pilot thinking). The phenomenon initially is 
given in vague terms, a general sense that becomes the seed to an evolving idea of 
whether or not a pilot assessed is proficient or what is the pilot’s level of knowledge, 
skill, or state. With an increasing number of observations, the sense tends to become 
more specific as it is increasingly concretized in documentary evidence on which the 
objective sense is based. There is a movement from abstractness to a concretely grounded 
sense, all the while there is a parallel movement from the concrete to the abstract, as 
the vague notion becomes increasingly structured and fine-grained. Ultimately the 
documentary method evolves an explanatory framework in every practical case. That 
is, the result of the documentary method covers everything, which is both its strength 
and its downfall, as one commentator on social psychology notes: “In any actual case 
it is undiscriminating and . . . absurdly wrong” (Garfinkel 1996). 

Some readers may be taken to think that the documentary evidence is the same as 
classical concept learning, where research participants derive concepts from instances 
and non-instances (Figure 2.1). Whereas there are some similarities, there also are 
some significant differences. In the classical case, the observations (facts) are clear, 
as there are only small, limited numbers of attributes; and the concept can be given in 
an unambiguous manner, such as “two circles or two boundaries” (Figure 2.1). In the 
assessment of pilots, however, the phenomena of interest themselves are fuzzy, as are 


+ s - + 

+ = = 
Figure 2.1 In the classical concept-learning paradigm, figures with geometrical shapes were used to 
constitute instances (+) and non-instances (—) of some concept that subjects were to learn. 
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many of the perceptual attributes (see Chapters 3 and 4). Although flight examiners 
talk about ideal performances, corresponding to the prototype of a concept without 
that such a prototype has to exist in any hard way. Ideal performances exist only as 
approximations, for even when the pilots examined are flight examiners themselves, 
the acting flight examiners highlight aspects that may be improved (see Chapter 3). 
Unlike what is apparent in the classical concept-learning paradigm, flight examiners 
tend to actively look for evidence of a certain kind or introduce specific failures that 
condition the kinds of problems that the pilots will face and the consequences of which 
the flight examiner will observe. In the documentary method, a cultural object (e.g., the 
concept of situation awareness) exists in and as the totality of evidence and, therefore, 
never is abstract. 

Existing research on pilot assessment notes considerable variability in the 
assessment scores used as part of a measurement paradigm (e.g., Flin et al. 2003). 
Flight examiners easily admit that assessing pilots is not a “hard science”. This 
understanding is captured in the notion of documentary sense. It goes together with an 
objective sense associated with flight examiners’ concrete observations that they take 
to be manifestations of some cultural object (e.g., decision-making skill or situation 
awareness). An analysis of sense shows it to be a contexture that is the basis for figure— 
ground distinctions, where the figure “makes sense” in relation to the given ground. In 
other words, any figure is a form of text that is tied into its surrounding context; and a 
text makes sense when there is a context into which it can be connected. The inter-rater 
reliability approach to human factors is based on the assumption that phenomena such 
as pilots’ knowledge, decision-making, management, or communication are objective 
phenomena that can be measured. When raters differ, problems are ascribed to the 
lack of rater training, the measurement instrument, or some other aspect. However, 
this chapter shows that the cultural objects are not themselves given. Instead, they 
are treated as invisible (like the contents of black boxes) but manifest themselves 
in some way. However, not all objective manifestations reflect what is taken to be 
the real underlying pattern. Moreover, the objective manifestations are not measures 
but constitute categorizations. That is, the method flight examiners employ in the 
assessment of pilot performance leads to a categorization rather than a measurement. In 
Chapter 3, the basis for this reconsideration is worked out to a substantial degree; and 
Chapter 4 shows that the process can be modeled mathematically and therefore is not 
so unreasonable at all. Instead, the process of assessment is consistent and principled 
within the subject considered (i.e., individual flight examiner or flight examiner pair). 
This is the source of the variations observed in previous research on pilot assessment 
in the method flight examiners use. But the variation is not random, as past research 
treated it within the measurement paradigm. Instead, flight examiners do have (and can 
give) good reasons for their sense that a pilot is or is not proficient—as can be seen when 
they collaborate in an assessment based on the think-aloud protocol part of Chapter 3. 

The documentary method allows understanding why flight examiners do not and 
cannot perceive all relevant facts (attributes) of an event, which mitigates how they 
rate the performance that can be seen. This is quite apparent from flight examiners’ 
assessments of one scenario involving an emergency landing following an engine fire 
(Weber et al. 2014a; Weber et al. 2016). All those flight examiners who did not notice 
that the pilots assessed evacuated the aircraft on the side of a running engine passed 
the crewmembers, but all those flight examiner pairs noticing this fact failed the crew. 
Whether the failure to observe such an important aspect would also occur during a 
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regular simulator session cannot be ascertained by the data available. Given the control 
the flight examiners have over setting up situations and their awareness of the flight as 
a whole (see above), such cases may indeed be rare—though in this study there were 
instances where flight examiners had missed important aspects of the flight, such as 
disabling the automatic pilot. 

Material phenomena are directly available such that they can be pointed to, taken 
in hand, or relatively agreed upon. Cultural objects—such as “Galilean pulsars”, 
Expressionism’, or simply “help”—are assumed phenomena available only indirectly: 
through their manifestations. The decision whether some observed (therefore material) 
fact is a manifestation of an underlying cultural phenomenon, a coincidence, or merely 
a contingency requires a methodical approach of one or another kind. As part of the 
documentary method, flight examiners use a variety of ways to increase the evidence 
for or against the existence of a phenomenon. Thus, for example, they select from a 
database of more than 200 incidences that might affect the flight in progress. They 
then conduct observations of the pilots’ performances in response to the disturbance 
at hand. In the end, they produce an assessment of the pilot or, in training situations, 
identify a collection of different issues that the pilots should focus on for the purpose of 
professional development. What the final narrative will be is unknown at the beginning. 
Yet every observation possibly has a place in the final story line, which is in part 
constituted by the observation history. Which observations will be included depends 
on the overall narrative as required by the plot, but the overall narrative depends on the 
observations (facts) made salient for the present purpose. The story is an emergent one 
and can change from one instant to the next in the case ofa serious performance issue. 

Some readers may be led to believe that flight examiners enact confirmation bias. 
This concept derives from classical hypothesis testing, which allows nothing to be 
concluded about the null hypothesis and only rejected alternate hypotheses constitute 
reportable findings. However, if the flight examiners’ method is to be compared to the 
sciences, then a more appropriate reference is Bayesian hypothesis testing. The classical 
approach to hypothesis testing is based on falsifiability, whereby the investigator states 
the null hypothesis and then, by means of experimental method, attempts to reject it. The 
logic assesses the probability p(data|H,). In this approach, efforts seeking to confirm 
of a hypothesis are taken to be a form of bias (confirmation bias). On the other hand, 
in the Bayesian approach the data play a different role. Data are used to update the 
probabilities of alternative hypotheses. The Bayesian approach therefore establishes 
the probability of one or more hypotheses p(Ħ |data). Each time new data are available, 
the probabilities of the hypotheses under considerations are updated yielding what is 
called posterior probabilities. If there are only two hypotheses, null (H,) and alternative 
(H,), then the two probabilities are directly related: p, = 1 — p,,,. Thus, a number of 
studies exhibiting no difference do constitute evidence for the null hypothesis. Flight 
examiners can be understood in terms of the creation, maintenance, and updating of 
multiple hypotheses. The probability of any hypothesis therefore is a function of the 
history of the investigation as ever-new data are brought to bear on it. In this approach, 
even some first observation has to be conducted with assumptions about the hypotheses, 
which, for statistical purposes and in the absence of other prior knowledge, are set to 
have equal probabilities. In the case of prior knowledge, it constitutes a starting point, 
which is equivalent to the situation where flight examiners make an observation but this 
observation comes with an interpretive frame, that is, one or another documentary sense. 
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This chapter also makes apparent the mutually constitutive relation between 
evidence and the phenomenon that it supposedly manifests. That is, for example, a flight 
examiner’s sense that a pilot has lost or diminished situation awareness derives from a 
particular observation; but this observation is made and explained by an assumed level of 
situation awareness (lost, diminished). The objective sense and the documentary sense go 
together and cannot be uncoupled. This therefore is not an example of simple observation 
bias, which would operate such that the observation is subjected to the presupposition. 
Instead, in the flight examiners method, the observation constitutes the documentary 
sense. This is similar to the findings of a study of classification where sociology 
graduate students were asked to code hospital records for the purpose of identifying the 
organized ways of an outpatient clinic that led to particular patient trajectories (Garfinkel 
1967). That study shows that the graduate students not only assumed the knowledge 
that their coding procedures were to reveal but also such knowledge was necessary to 
make decisions about what really happened in the outpatient clinic. Studies show that 
seasoned experimental biologists used this same method while attempting to interpret, 
understand, and explain their data and the associated graphical representations (Roth 
2014); and so do qualitatively working social scientists employing the highest levels 
of rigor (Roth 2015d). Thus, the documentary method of interpretation differs from 
testing (given) hypotheses, because the cultural object (proficiency, knowledge, skill, 
state) is itself a function of the observations. It differs from confirmation bias, observed 
for example in people reading the horoscope, in that the flight examiners create the 
explanatory frame based on the data. The equivalent case—if such can be made—would 
be people trying to figure out what their daily horoscopes say based on their concrete 
experiences without ever having access to the horoscope. 

There is a temporal order when flight examiners work in the simulator, where they 
only have “one shot” at making observations in any one instance. If they miss a real 
fact, it will not and cannot enter the overall story line. When there is an opportunity 
for replaying at least some episodes, such as with the debriefing tool or in the case of 
the modified think-aloud protocol, facts may be discovered after a first, second, or later 
viewing. This was the case with the failure to push the go-around button, which flight 
examiners may notice only after repeated viewing (see “Missed Approach” box, Chapter 
1). There is therefore an emergent sense of what the narrative might be. After an initial 
assessment has been made, the results can be revised, such that an initial pass (perhaps 
with markers) might turn into a fail. Less frequently, flight examiners wait to find out 
from the pilots what they have to say to a critical incident before assessing a particular 
event (see Chapter 7). Although rarely observed, flight examiners do take up and take 
into account what they learn about some event into their assessment of this event. This 
is especially so when the debriefing tool is used to replay events. What has happened 
as seen in the videotape is taken as the way, that is, as objective evidence that overrides 
what either pilots or flight examiners have remembered prior to viewing the video. 
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+ The reasoning underlying performance assessment is social through and through. 
Because reasons inherently are produced for others, to be intelligible to others, 
and oriented to solicit actions from others, they are cultural rather than individual 
features. 


e Performance assessment changes in form and content with experience. Pilots less 
seasoned in performance assessment tend to follow the structure of the assessment 
instrument (when available), which also shapes what they look for and how they 
look at evidence. Seasoned flight examiners orient towards story lines that describe 
and explain the performance of pilots. 

e The numerical ratings of pilot performance vary widely, even when assessors 
work in pairs where they are forced to make reasoning public and explicit for the 
purpose of reaching agreement. 
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+ The same micro-event, such as a pilot’s statement about the weather and the actions 
that follow, may be used as objective evidence for constituting a documentary 
sense of a different human factor. As a result, the same facts and micro-events are 
explained by different human factors. 


e Within different interpretive frames (horizons), assessors perceive the same actions 
and talk in different ways, with considerable consequences on the assessment and 
the ultimate decision whether a pilot should be passing or failing an examination. 


+ Flight examiners produce substantially more gestures symbolizing aspects of 
pilot performance and aircraft configuration than captains and first officers. These 
gestures also manifest the holistic nature of flight examiners’ knowledge of flying 
and performance assessment. In the constitution of reference performances against 
which pilot performance is evaluated, flight examiners do not recall individual facts 
but verbally and gesturally produce sequences representative of kinetic melodies. 


xamining pilots is a form of expertise. There now exists a long history of studies 

of expertise in very different domains focusing on such issues as problem solving, 

decision-making, and categorization in complex, real- and micro-world systems. 
Many studies focus on differences between experts and novices. A much smaller number 
of studies focus on expert/expert differences, for example, in (a) chess players at the 
international masters versus class A levels, (b) historians’ categorization of historical 
texts, (c) English professors’ interpretations of academic arguments about poems, 
(d) biologists designing experiments, or (e) public sector versus university biologists 
reading/interpreting graphs. In all such studies, the research investigates, assesses, and 
evaluates the real-time performances of experts both within and out-of their field by using 
a think-aloud protocol.! Much less, if at all studied are the cognitive processes involved 
in assessing the performances of peer experts and the form and quality of reasoning 
underlying performance assessment. This includes aviation, which means that there 
exists little research on the cognitive processes of flight examiners—who themselves 
are experienced pilots—in the process of assessing pilots. To better understand changes 
in the ways flight examiners rate the performances of peers, it is useful to examine in 
some detail novice and seasoned raters. If participants are chosen among pilots who are 
not yet flight examiners but potentially will be in the future, the research may reveal 
changes in rating quality and reasoning at different points of the career trajectory of 
those pilots who eventually become flight examiners. 

This chapter is concerned with understanding performance assessment-related 
forms of reasoning in the culture of aviation generally and with how pilots mobilize 
performance categories in the course of assessing performances of peers. However, 
rather than searching for reasoning in the privacy of individual minds, the phenomenon 
is investigated ethnographically by looking at situations in which pairs of pilots together 
assess other pilots. When such pairs are required to produce one assessment for each 
pilot, individual participants are forced to articulate their reasoning for one another. 
Their reasoning thereby also is made available to the investigator. Moreover, those very 





' Tn the think-aloud protocol, participating experts say aloud anything that is on their mind while doing 
a field-relevant task. 
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forms of reasoning that pilots exhibit to others in their joint work are the same when 
individuals do performance assessment on their own—even if they are not required 
to produce a record to be kept on file. Because becoming a flight examiner is part of a 
career trajectory that begins with learning to pilot an aircraft, eventually becoming first 
officer on a specific aircraft type before upgrading command to captain, this chapter 
investigates possible differences when pilots of different ranks (first officers, captains, 
and flight examiners) assess the same videotaped scenarios. Ultimately such work 
should help the aviation community to better understand the development of flight 
examiners, which occurs through the individualization of cultural-historically situated 
assessment practices. 


PERFORMANCE ASSESSMENT 


There exists considerable research with respect to the different dimensions of 
performance assessment for those in high-risk fields such as medicine, maritime 
industry, and aviation specifically. The dimensions of interest include the conditions 
under which individuals or teams are assessed and the authenticity of performance in 
simulators, the instrument used during the assessment process, how to train those tasked 
with assessing performance, and how assessors make judgments about performance. 
Here the focus is on performance assessment conducted by peers in aviation, a context 
where peer assessment (evaluation) is part of the regular and mundane work of those 
involved. Experienced pilots, serving as flight examiners (“check pilots”), draw on their 
expertise as pilots and some additional experience during the initial steps of becoming an 
examiner to assess the performance of other pilots. At least one theoretical model makes 
an explicit link between evaluative skill and expert performance (Weiss and Shanteau 
2003). In this model, (a) evaluative skill plus qualitative/quantitative expressive skill 
yields expert judgment, (b) evaluative skill plus projective skill yields expert prediction, 
evaluative skill plus communicative skill yields expert instruction, and (c) evaluative 
skill plus execution skill yields expert performance. However, there is little reason to 
expect positive correlations between years on the job and performance (Shanteau et 
al. 2002); and there is evidence that a correlation between rank and expertise is not 
always positive. 

Performance assessment in aviation specifically has been suggested to move from 
categories (e.g., cooperation), collected into to elements (e.g., considering others, 
supporting others), to result in a pass/fail rating (O’Connor et al. 2002): categories 
> elements > pass/fail. However, to make an assessment with respect to a particular 
element, an examiner of pilot performance draws on a specific observation—made 
in real time, recalled using a simulator-mounted video-recorder (debriefing tool), or 
pre-recorded in a scenario—that becomes a fact or stimulus. For example, an examiner 
might note that the instrument approach plate calls for some flight path, altitude 
constraint, or speed limitation; but actual instruments indicate the aircraft is deviating 
(momentarily) from the required parameters. This may be taken as an indication that 
there are “minor lapses or mistakes in monitoring”, where monitoring is an element 
in the category “Aircraft maintained within tolerances”. We may therefore expand the 
assessment process in this model to include facts/stimuli: facts/stimuli > categories > 
elements > pass/fail. 
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The categories used in performance assessment require shared defining 
characteristics, such that, for example, doctors have to agree what a myocardial 
infarction is. However, the identification of specific symptoms among a range of facts 
requires perceptual and integrative skills. That is, expert judgment is subject to two 
essential—necessary but non-sufficient—criteria: discriminative ability, which serves 
to distinguish between similar but non-identical aspects of the case to be assessed, 
and internal consistency. When examiners of pilot performance assess pilots shown in 
video clips, they typically view complex performances with a duration much less than 
a typical LOFT—e.g., between 3:30 and 9:20 min in one series of studies (Roth and 
Mavin 2015; Roth et al. 2014b; Weber et al. 2016). In contrast to observation during 
LOFT sessions, those assessing pilot performance shown in video clips tend to have 
all the time they require for identifying specific performance aspects and flight-specific 
facts—e.g., speed, altitude, visible and aural alarm signals, rate of descend, banking 
angle, and so forth—to evaluate the performance and produce an assessment based 
on supporting evidence. That is, a lot of time is available for establishing an objective 
sense on which the documentary sense of the different cultural objects (human factors 
and other acceptable concepts) is based (see Chapter 2). Most importantly, even when 
experts agree, they may still be wrong; and critical facts that mitigate performance may 
be hard to detect or experts may notice different facts or, when identifying the same 
facts, interpret or weigh them differently. 

When individuals make global performance assessments, they make them 
from cues or facts associated with the area being judged, though these cues are not 
always considered equally. This global judgment is based on the documentary sense 
(Chapter 2) and derives from component judgments (e.g., “situation awareness”, 
“management”, or “decision-making”); and the documentary sense constitutes a domain 
expert’s combination of the components. Each component itself is derived based on 
the documentary sense, which in turn is determined by means of the objective sense 
(see Chapter 2). Cognitive combination of components is not always accomplished 
in a predictive way, and experts, to simplify the problem, may not even consider all 
components in the way they would if they were using a linear model. These expert 
combining rules constitute non-compensatory decisions. That is, certain components 
must be present and are non-negotiable (non-compensatory); non-compensatory models 
do reflect expert judgments. Performance assessment therefore may be constituted 
by the sequence: facts/stimuli > categories > elements > combining rules > pass/fail. 

As in the case of the performance assessed, the quality of performance assessment 
changes with the number of assessments made. For example, trained raters of writing 
samples tend to be more reliable than raters with little or no training. As novice raters 
increase in the number ratings completed, their assessments align with those of expert 
raters of performance (Lim 2011). The differences of novice raters with respect to 
seasoned raters may be to either side—much harsher or much more lenient at the 
outset. Initial variance in the consistency of ratings seems to rapidly disappear with 
number of ratings completed. There are suggestions that high levels of rater agreement 
can be achieved even among novices as long as they are exposed to joint coding with 
other raters working on the same data (Schoenfeld 1992). Subjective judgments of 
performance are exposed to the incentive structure, which means that possible bias 
exists with respect to how assessors are monitored and assessed. In a culture of peer 
performance, it may be possible that lower-ranked individuals systematically assess 
performance more positively when the rated individual has a higher rank. 
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Peer assessment exists in many fields, such as the military (promotions), academia 
(tenure, promotion, peer review), nursing, or aviation. In aviation, senior pilots (referred 
to as check captains or flight examiners in the Asia Pacific and as type-rated examiners 
in Europe) conduct the bi-annual training and performance assessment of pilots. These 
senior pilots continue to fly but, in addition to their regular job as pilots, also conduct 
the performance assessments required by the regulatory authorities. Each pilot may 
be identified with points along two dimensions of expertise—as pilot and as assessor 
of piloting performance. 

Performance assessment is integral to the career trajectory of a pilot. From 
the day pilots begin with an airline they will be assessed—in some airlines at least 
four times per years—until retirement. First officers are promoted to captains when 
flight examiners have assessed their performances as satisfactory. This supposes that 
captains have demonstrated sufficient levels of expert performance, and, therefore, both 
evaluation skills and execution skills. Captains become training captains (instructors), 
which presupposes that they exhibit more than entry level of communicative ability 
that accompanies evaluative skills. Some training captains become flight examiners, 
which supposes that they exhibit a considerable degree of expert judgment. Years on 
the job is often used as a proxy for expertise, but the number of years does not tend to 
correlate well with rater accuracy—although the confidence increases with the number 
of assessments conducted. In the airline industry, the number of flight hours generally 
and the number of flight hours on type specifically are better proxies for extent of 
experience and practice, and, therefore better proxies for expertise. 

In the aviation culture, examiners of pilot performance are peers rather than 
practice-removed evaluators. It has been suggested that performance assessment culture, 
based on peer evaluation, may be based on folk models (Dekker and Hollnagel 2004). 
These authors include two of the categories used by pilots in the present chapter in a 
folk model about human performance. Thus, the category of “situational awareness 
is ‘too neat and holistic’ in the sense that it lacks such a level of detail and so fails to 
account for a psychological mechanism needed to connect features of the sequence of 
events to the outcome”. The purpose here is not to pursue the nature of the categories 
and their elements that the pilot assessors of piloting performance use to compare them 
to scientific categories (see Chapter 1). Rather, this chapter describes how examiners 
of pilot performance use the performance categories that are familiar to them. They 
use them in conversations with other examiners and, thus, arrive at shared uses of the 
performance categories. Because reasoning always is for others and in fact has its genesis 
in relations with others (e.g., children and their parents), it is a social phenomenon in the 
strong sense (e.g., Livingston 2008). That is, reasoning is social even when individuals 
reason when they are by themselves. This is so because in the case of private cogitations, 
these reasons are provided as to another in the self: “I relate to myself as people related 
to me” (Vygotsky 1989). 


SCENARIOS AND ASSESSMENT 


The taped scenarios used in this research provide a lot of visual information to the 
raters, who watch the videos on 52-inch monitors. There are close-ups of the instrument 
panels, overview shots of the left or right sides, overview shots of the cockpit as a whole, 


76 Cognition, Assessment and Debriefing in Aviation 


and approach plates. Although such videotaped scenarios do not have the same high 
fidelity that an assessment in a simulator or aircraft would have, the scenarios used 
provide sufficient information for the raters to make and justify their ratings. Because 
the research involved pilots from two airlines flying two-engine propeller aircraft from 
two different manufacturers, the scenarios were produced for both aircraft types and 
relative to flight routes and airports that they were familiar with. 


Scenario 3.1: Go-around on Circling Approach 


The first scenario used in this research is the Missed Approach situation described in 
Chapter 1. Here, a brief recapture is provided. Poor weather, consisting of low cloud 
and rain, is simulated at the airport. To accomplish a safe landing, pilots are required 
to fly a specific instrument approach, and for this scenario, a circling approach is 
conducted (Figure 1.1). The circling approach has the aircraft descending safely on 
a predetermined route, using cockpit instruments to a specified altitude, called the 
minimum descent altitude (MDA). Once reaching this altitude, if visual reference of 
the runway is obtained, the pilots are to maneuver (called circling) the aircraft around 
the airport while keeping it in view, and prepare the aircraft for a landing on the chosen 
runway. If visual reference to the runway cannot be maintained, which tends to occur 
when entering instrument meteorological conditions (IMC) such as rain, low cloud or 
snow, then the pilots conduct a missed approach and, flying a go-around procedure, make 
the aircraft climb back to a safe altitude. The circling approach differs from a runway- 
aligned approach where visual maneuvering is not required. The circling approach is, 
as several participants noted, possibly the most hazardous kind of approach especially 
at the chosen airports with mountains to the right hand. 


Scenario 3.2: Limited Fuel Resources after Go-around 


In the second scenario, the weather conditions are such that at the missed approach 
point, visual contact with the runway is not established. This leads to a missed approach 
call followed by an ascent toward the missed approach altitude. Already during the 
approach, the pilot flying (first officer) has commented on the low fuel situation. 
Following the missed approach but prior to reaching the missed approach altitude, the 
crew discusses the available options. The captain proposes flying to the turning point 
(as specified in the standard operating procedures) and attempting another approach 
to the current destination airport. The alternate airport would be the option in case a 
second missed approach was necessary. From an economic standpoint, landing at the 
current airport is preferable. The first officer, on the other hand, states a preference for 
immediately going to the alternate airport. His reasons include the currently existing 
headwind, limited fuel resources, and marginal weather conditions even at the alternate 
airport. In this case, the first officer is outlining a concern about the aircraft being placed 
in a situation where it may go below normal fuel reserves. The main issues identified 
include the compromising of fuel reserves inherent in the captain’s proposal and the 
degree to which he attempts to convince the first officer to accept it. 


Scenario 3.3: Engine Fire before Landing 


During the approach, after the aircraft is already configured for landing (flaps set, landing 
gear down), a fire occurs in the left, number | (left) engine. The pilots shut down the 
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Figure 3.1 The reading of the propeller (Np) instruments for left (no. 1) and right (no. 2) engine in 
the engine fire before landing. 


engine and follow existing procedures for extinguishing the fire and continue with the 
approach. They make a mayday call. The pilots inform cabin crew before touch down 
at an altitude of 500 feet and then land the aircraft. On the ground, after having received 
the clearance from ATC and with the aircraft still on the runway, the captain makes a 
PA asking passengers to evacuate to the right-hand side. At that time, the propeller (NP) 
gauges on the engine display are configured as in Figure 3.1—whereas the left propeller 
has stopped spinning, the right propeller still operates at Np = 80%. 


Assessment Form 


Together with their consultant, who originated the performance model (see the critique 
of the MAPP model in Chapter 1), several airlines have developed and use an explicit 
performance assessment instrument (Roth et al. 2014b; Weber et al. 2014a; Weber 
et al. 2014b). In the instrument, each of the six main performance categories of the 
performance model is broken down into subcategories: communication (clear and 
correct, information flow, timeliness), management (workload, control, cooperation, 
threats & errors), knowledge (facts, procedures), aircraft maintained within tolerances 
(manual flight, automation, monitoring), decision-making (time, facts & diagnosis, 
option generation, risk assessment, plan & contingency), and situational awareness 
(perception, comprehension, projection). On each of the 20 resulting subscales, a 
pilot's performance is rated from a score of 1 (very poor) to 5 (very good). Based on 
the subscale score, assessors then determine an integer composite score for the main 
category. In airlines using this instrument, pilots automatically fail when they receive a 
1 (very poor) on any single main category. A fail rating also results when three or more 
main categories are rated at the 2-level (Mavin and Roth 2014b). To aid assessors, the 
instrument provides verbal descriptions of the level of performance corresponding to a 
particular score (Table 3.1). Thus, for example, a score of 1 on the situation awareness 
“projection” subscale corresponds to the description “did not predict future events, even 
those obvious to flight safety” whereas a score of 3 corresponds to a situation where a 
pilot is seen as displaying “some difficulty predicting future events”. All participating 
pilots have received training in the use of the instrument during their CRM training, 
where they rated pilots in videotaped scenarios. They had opportunities during the 
CRM course to discuss ratings with others. Moreover, all pilots have been evaluated 
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in the airline using this instrument; and, in the course of the year preceding this study, 
all flight examiners have used the instrument to assess pilots during bi-annual recurrent 
training and assessment. 


HOW PILOTS AND FLIGHT EXAMINERS ASSESS PERFORMANCE 


In Chapter 6, a model is presented for the order of the world articulated in (a) praxis 
(mimesis, ), (b) theory (mimesis,), and (c) application of theory to praxis (mimesis,). In 
assessment, an abstract description of pilot performance is produced. This description 
exists in the textual world of mimesis,, where “text” is taken broadly to mean any form 
of verbal, mathematical, and diagrammatic representation. In the measurement processes 
of the natural sciences, engineering, and technology, a measurement instrument may be 
held directly to the phenomenon, such as when a ruler is held to an object. However, 
when the phenomena to be assessed are human actions, a first reduction has to occur 
by means of which the complex transactional world is translated into separated actions 
and interactions attributable to agents and their relations (e.g., Schiitz 1932). Even the 
videotape already is a form of representation that no longer conforms to the irreversibility 
of actions in a once-occurrent world (Ashmore and Reed 2000). Videotapes can be played 
over and over again, thereby supporting the extraction of facts and phenomena that 
may be nearly impossible to extract when real-time witnessing is the only observational 
mode available. Moreover, videotapes constitute a point of view, which always already 
is partial and does not represent the totality of the lived situation. The very evaluation 
of individual pilots—as if they were elements from which a complex joint cognitive 
system is built—presupposes the translation from the transactional world (where cause 
is a function of effects) into a world of text that is about events already completed and 
lying in the past, and where cause and effect ex post facto are separable and are related 
sequentially. 

Actions, always in flux and changing, become objective in a process of fixation 
that makes them a thing. Any action may then be treated in the same way as a verbal 
utterance in speech act theory (Ricoeur 1991). This theory distinguishes between the 
performance (locutionary act), intention (illocutionary act), and effect of an action 
(perlocutionary act). The performance has a propositional character typical of the 
sentence structure of Western languages, with subject, verb, and object. The subject is 
the agent, who, based on some intention, acts upon the object, which may be another 
being. In acting, the agent brings about an effect. When actions have this propositional 
character, they immediately reveal their source and the underlying intention. 

This process of objectification is at work in all forms of assessment that may be 
observed in aviation. Two general types of assessment process can be seen when pilots 
are observed assessing other pilots while drawing on an assessment form (see box 
Evaluating the Missed Approach). Process A begins with picking a verbal description 
from the assessment form and seeking/identifying evidence for or against it. Frequently, 
assessors then take the corresponding verbal description in the column to the left and 
right (e.g., Table 3.1) and check its fit with the videotaped event. Here, the assessors 
take the propositional structure available in the verbal descriptions of the assessment 
instruments and then seek to identify instances in the videotape that are consistent with 
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the description. The propositional structure is given, and the world (scenario) is then 
fit to an already existing (textual) order. 

In Process B, pairs first construct a narrative of the whole scenario. Only when 
a satisfactory description of events exists do performance assessors then ask how the 
narrative constructed maps onto the behavioral descriptions provided by the assessment 
instrument. The main concern therefore is not the assessment score or a pass/fail rating. 
Instead, the main concern is the establishment of a plausible and coherent overall 
story—which is why good flight examiners are sometimes likened to good storytellers 
(see Chapter 2). Process B thereby amounts to finding a propositional structure that 
captures the sense of the emerging story as a whole, thereby contrasting Process A 
where the performance is looked at through the lens of a given propositional structure. 
In Process B the story renders a particular action reasonable, so that the narrative as a 
whole becomes a constraint of the specific actions selected as objective evidence (fact). 
Readers may find it helpful to use the analogy of a detective charged with figuring out 
what precisely happened and who committed the crime. Here, too, the scene of the 
crime provides a lot of facts not all of which are relevant to the crime. But the detective 
does not know beforehand which facts are and which are not relevant. The overall story 
makes certain facts relevant, but the facts determine the overall story. In Process A, 
however, a pre-established scheme is used to pick out facts to go with one or another 
rating. The two processes therefore differ in the way they fixate actions. 

When there is a question regarding facts, participants frequently return to the 
videotape, navigate to the pertinent sequence, and ascertain the response to their 
question. For example, if a question arises about the engagement of the autopilot 
following the beginning of the go-around procedure, which observers may have missed 
perceiving during the first pass through the videotaped scenario, then they can and do 
return to the videotape. Assessors return to the video to seek to verify whether or when 
the go-around button is pressed or whether and when the scenario pilots talk about 
the autopilot. Exemplifying scenarios for the two processes are provided in the box 
“Evaluating the Missed Approach”. 


Evaluating the Missed Approach 


A number of studies have investigated how pilots of different rank assess the pilot 
performance in scenarios like the Missed Approach (Roth and Mavin 2015; Roth 
et al. 2014b; Weber et al. 2016). In one part of the research, airlines had a model of 
pilot performance using human factors terms for technical skills (knowledge and 
procedures, aircraft flown within tolerances) and non-technical skills (situation 
awareness, decision-making, management, and communication). Aircraft flown 
within tolerances, decision-making, and situation awareness are grouped together 
in the category essential skills, whereas the remaining factors are grouped into a 
category of enabling skills. The airlines use an evaluation form in which scores for 
the level of performance (from 1 [very poor] to 5 [very good]) is mapped against 
corresponding verbal descriptions. Thus, for example, the main category of situation 
awareness includes three subcategories: perception, comprehension, and projection. 
Levels of each of these subcategories are described in a sentence, such as “lacked 
awareness of clearly obvious systems or environmental factors”, which corresponds 
to ascore of 1; the sentence “perceived all systems or environmental factors affecting 
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flight” corresponds to a score of 4 (Table 3.1). To arrive at a score for the main 
category, assessors used the trend available in the three subcategories. Thus, two 
scores of 2 and one score of 3 for the subcategories yield a 2 for the main category. 
In the airlines using this system, three main categories with a score of 2 or a score 
of 1 on any main category meant the examination was failed. In other airlines, no 
explicit human factors model existed. 


Process A: Instrument-Driven Performance Assessment 


In one approach to the assessment task, raters, following a brief description of the 
overall situation, take the assessment form and work their way from top to bottom in 
sequential order. Within each category, a rater would begin somewhere in the middle, 
read a “word picture” and then find evidence for or against it and then went on to 
another word picture. The process continued until they found and circled the one 
that appeared to best one for describing the performance of the pilots in the video. 
The following exemplifies how assessing a pilot would work in general. 


I go to the middle of the ratings and then find out whether to go left or right. 
Okay, “Perceived significant systems or environment factors affecting flight” 
(Table 3.1, SA1.3, i.e., row 1 column 3). Actually, the pilot flying did miss 
some factors, like what the minimum descent altitude is or in which direction 
to turn after calling the missed approach. These were the two gaping holes 
in the performance. So going to the left: “Missed some minor systems or 
environmental factors not critical to flight safety” (Table 3.1, SA1.2). The 
direction to take after the missed approach call is more than a minor factor: it 
is a major factor. But perhaps not knowing the minimum descent altitude may 
indeed only be a minor factor. Also, the pilot flying missed that there is high 
terrain, a mountain range to the right. The pilot must not turn into this direction. 
So one may have to consider going further down. “Lacked awareness of clearly 
obvious systems or environmental factors” (Table 3.1, SA1.1). Itis probably that. 
Now to “misinterpreted or did not comprehend factors affecting flight safety” 
(Table 3.1, SA2.1). At first, the pilots comprehend external factors, like when 
anticipating that they have to watch out because of the clouds in the circling 
area. But when they have to conduct the go-around, the performance of the 
pilot flying just goes from bad to worse. The situation awareness completely 
disappears, and quite rapidly so. There is also the issue with the wind. When 
they do the circling approach in that situation, they will have a strong tailwind; 
or maybe the wind is coming across. So does the pilot flying comprehend the 
issue of the cloud at circling or the wind? Perhaps he notes them initially, but 
then he does not follow through. The way in which the crew acts is as if they are 
surprised, which means, they have not taken into account their future actions, 
“even those factors obvious to flight safety”, what they initially articulated. 
So it probably is a “1” on projection (Table 3.1, SA3.1). But because they do 
comprehend the factors and they do note that they have to pay attention, they 
in fact comprehend those factors at that time. 
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Process B: Getting a Holistic Picture First 


The more seasoned flight examiners, pilots frequently with more than 10,000 flight 
hours, tend to go about the assessment tasks differently. They spend a lot of time 
building a case description, considering all aspects of the scenario. They often 
form a first conceptualization, and then replay all or parts of the scenario, using 
the information garnered to augment, elaborate on, or correct earlier parts of the 
narrative plot they are constructing. Then, once they have arrived at a sufficiently 
detailed description, these seasoned flight examiners move rather quickly through 
the assessment form assigning the scores based on the description of the performance 
that they have previously built. 


The pilot flying has some issues with situation awareness. While the aircraft 
is going downwind, the pilots are discussing the fact that the weather is not so 
crash hot in the area where they will be circling. So at this point they are aware 
of the situation; they discuss it then and there. Initially, they are visual; they 
can see the airport. They also can see what the weather is like in the circling 
areas and that they might go IMC downwind. They could have decided at this 
point to do a missed approach, which would have made their task easier. When 
they in fact continue on their approach, they should have gotten ready and be 
prepared for the IMC. But then they go IMC while turning onto left base; and 
it looks as if they are surprised. At that point, they still have the inbound course 
set, which sets them up for screwing up the missed approach. They are rolling 
away from where they should be going. The pilot flying says that it is a right 
turn, which it is not. It should have been set as an outbound course, because he 
is doing the circling approach. But the navigation aid still is set as it was on the 
inbound course. So it gives him the reverse information, turned by one hundred 
and eighty degrees, in fact, opposed to what the pilot should be getting. The 
demands on situation awareness build up from there, especially as they near 
the missed approach situation. This is why they get opposite the direction for 
where to steer. It is at that point that everything just blows out of proportion. 
When the turn is announced to the right, and the pilot monitoring has to help 
out, their situation awareness disappears. Taken together, all of this amounts to 
a failed examination. The PF has to do the exercise again. He has to repeat it. 











Process A: Instrument-Driven Performance Assessment 


The performance assessment process observed when the assessors are not already flight 
examiners tends to be driven by the structure of the assessment instruments (see box 
for an exemplifying scenario). The assessment process takes one major assessment 
category at a time for the purpose of arriving at a score. The overall assessment then 
results from considering together the scores for the six categories that the instrument 
contains. As noted above, according to policies of the airlines using this assessment 
form and where the research was conducted, a score of | on any one of the six categories 
means the examination is failed; a fail rating also results when three or more scores 
of 2 have been assigned to any subset of the six categories. The following description 
of the complete assessment of the situation awareness category exemplifies how the 
structure of the instrument affects the process as a whole. Thus, in the same way as 
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the notebooks and debriefing tool structure the work of flight examiners (see Chapter 
5) and in the way the tools available in the debriefing room mitigate the event and the 
learning process (Chapter 8), the availability of an assessment instrument has an effect 
on the process and the content of performance assessment. 

Without much ado and after having done a brief recapitulation of the event shown 
in the videotaped scenario, assessors employing the assessment form-driven process 
begin orienting somewhere on the form. For example, they may begin by focusing 
on score 2 for the perception subcategory (row 1) under situation awareness (SA1.2, 
Table 3.1), read the verbal description (“Missed some minor systems or environmental 
factors not critical to flight safety”). They then identify the fact that the pilot flying (PF) 
misses the minimum descent altitude (MDA) and the direction to fly on the missed 
approach—this is important because there is high terrain to the right of the aircraft 
and hence the necessity to turn the correct way. As the Process A-related scenario (see 
box) exemplifies, assessors consider looking in the column corresponding to a lower 
rating (SAx.1). Following their reading-aloud the description concerning the lack of 
awareness of clear and obvious systems or environmental factors, the fact comes to be 
noted that the wind direction from behind while the aircraft was tracking toward the 
final turn (see Figure 1.1) would require attention given the circling approach (SA1.1). 

As in the scenario for Process A (see box), assessors work through the bulleted 
items in the second column of the situation awareness category. In the course, they 
assert that knowing MDA is critical, thus contradicting the “not critical to flight safety” 
(SA1.2, Table 3.1). They assert that the PF is not just acting, but checking with the pilot 
monitoring the desired MDA and the way in which to turn for the missed approach. 
Concerning projection, assessors first note that “difficulty predicting future events” 
(S3.2) is not so obvious. They state that the aircraft was correctly configured, appropriate 
airspeed, but note that the missed approach altitude is set relatively late. This is used as 
evidence for the claim that the missed approach was not predicted. Assessors note that 
pilots would like to see a limiting altitude set by the scenario pilots, and the failure to 
do so is taken to be evidence for the fact that the PF has “difficulty predicting future 
events” (SA3.2). The assessors confirm that it is the job of the pilot flying to set the 
missed approach altitude. They note that the PF is under high workload, flies visual 
in a circling approach low over the city, and has the autopilot on. On this basis, they 
“head down” to the first column, eventually circling “lacked awareness of clearly 
obvious systems or environmental factors” (SA1.1) as the description best matching 
the observed performance. Once done with discussing situation awareness, based on the 
verbal descriptions circled, the decision is made that on this dimension alone, the PF 
should fail and should do the exercise again. The movement of the assessment process 
thus may be described abstractly in this way: 


SA4 > SA3 > SA1.2 > SAl.1 > SA1.2 > SA2.2 > SA3.2 > SALI 


From orienting toward a situational awareness rating of 4, there is a reorientation 
toward the rating of 3, followed by attending to the verbal descriptions of individual 
subscales. 

In this manner, assessment moves through all six major assessment categories. 
Indeed, the assessment process follows, top to bottom, the order of the categories 
on the assessment form—i.e., situation awareness, tolerance, decision-making, 
knowledge, management, and communication. As a result, the assessment comes to 
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be pieced together by considering the videotaped pilot performances through the lens 
of the instrument, by assessment categories. The performance of each scenario pilot 
is considered as pertinent to the performance categories and in a pre-chosen order 
rather than in terms of a holistic image, as this occurs when flight examiners use the 
documentary method of interpretation (e.g., Chapter 2). 

This process of assessing pilots essentially begins with a set of descriptions and 
then seeks to identify that set that provides the closest match with the object (here video) 
looked at. In this, the process resembles the use of the Munsell chart employed in the 
natural sciences and anthropology for determining as exactly as possible the color of 
an object. Each page of the chart contains viewing holes surrounded by standardized 
colors. Scientists look at their sample—soil, lizard, etc. (Goodwin 1994; Roth and Bowen 
1999)—through the holes in search for the one where the surrounding color provides the 
best match with the sample. The area surrounding each hole has the function of a screen 
that blocks out a part of the perceptual field immediately surrounding the specimen. 
Whereas this increases the reliability of the measurement, it also decontextualizes the 
color sample: it no longer matters whether there is a clump of soil, a lizard, or something 
else as long as the color matches. The pilot performance assessment instrument functions 
in the same way, when assessors take a description, such as “Missed some minor systems 
or environmental factors not critical to flight safety” (SA1.2, Table 3.1) and then ask, 
“Does this match what is seen in the videotaped scenario?” To affirm or disconfirm, 
assessors might hold the description against different parts of the video they have seen. 
If there is a match, then, the score is determined. In the present example, it is a rating 
of 2 on the perception subcategory (i.e., SA1.2). This analogy with the Munsell chart 
makes apparent that the assessment process is not a measurement but a categorization 
whereby the assessors’ work and cognition exists in determining the specific category 
(hole) into which the observed performance fits. 

The resulting assessment of the pilot is given by a combination of descriptive 
statements. In essence, therefore, the order of the world seen in this way mirrors the 
order of the text constituting the instrument. The order in and of the textual world 
(mimesis,, see Chapter 6) is imposed onto the world of the scenario pilots. In the case 
of the videotaped performances, the assessment is performed no longer on the real world 
with a first-level order apparent in the performance (mimesis,) but on a represented 
world already viewed through a particular lens, which has been arrested in the many 
images that make a movie reel that can be played over and over again. The “chasm 
between lived act and the ‘same’ act’s representation” (Bakhtin 1993) has already been 
crossed and the assessment is in fact an assessment of a representation. 


Process B: Getting a Holistic Picture First 


In the second process—characteristic of seasoned flight examiners—a holistic picture 
in the form of a story is constructed first before assigning scores to any individual 
performance category. As the scenario shows (see box), the assessment is concerned 
with narrating the events in such a way that a big picture emerges and a consideration 
of the pilot as a whole. For example, rather than merely focusing on the question about 
the right-left confusion at the beginning of the go-around procedure, flight examiners 
may take into account the entire history of the flight. As the scenario shows (see box), 
an attempt is made to understand how the aircraft gets into the situation where it is; and 
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it is in that historical consideration that the attempted right turn makes sense. The overall 
account shapes the propositional structure of the actions: the right turn is understood as 
the result of a historical development in which the aircraft “tracks the radial the other 
way around”. In the plot of that overall story, the PF is not aware of the way in which 
the aircraft “tracks out” the radial in the opposite sense. 

Following the building of a case scenario in their own words, examiners then 
shift to the assessment instruments, read descriptive statements, and work backwards 
from there to the case that they have been narratively building. They may note that the 
communication of the scenario crew was right on the mark. Examiners may note that 
this is the case, and therefore of special importance, even when the PF is confused, 
announces the confusion, which allows the problem to be captured and dealt with. 
Seasoned examiners may ask whether the PF can be certain that the PM is correct in 
calling for a turn to the left. Thus, overall, communication may be assessed positively, 
as being nice and clear. Examiners may note, however, that the communication has not 
been timely, documented by the fact that at one point in the scenario the captain asks 
for the flap setting although the first officer is waiting for the response from air traffic 
control. Flight examiners may turn to the assessment form to seek assistance and read 
out the verbal descriptions corresponding to the scores of 3 and 4, respectively. Timely 
communication would be consistent with the observations, for there is only one small 
problem identified. Here, therefore, there is a general accounting of communication over 
the course of the entire scenario and the specific rating becomes an issue only at the 
very end, almost as an afterthought. The considerations are oriented toward the general 
features of a narrative account rather than organized around the verbal descriptions 
available in the assessment instrument. During the actual rating, flight examiners may 
refer to a previous part of the session where all the communication-related facts have 
been articulated in greater detail. 

When assigning scores, flight examiners follow an order of the categories in the 
way that these appear or are related in the depiction of the formal assessment model. 
That is, they organize the assessment conceptually rather than according to the linear 
structure of the assessment instrument itself. Thus, they begin at the bottom of the 
hierarchy and with the collection of the “enabling skills” (communication, management, 
and knowledge); on the assessment instrument, this corresponds to going from the 
bottom upward. In most instances, flight examiners continue the movement toward the 
“essential skills”, first covering those in the second level of the formal performance 
model before dealing with situation awareness. But some flight examiners may in fact 
be following the direction of the arrows on the formal assessment model, which point 
upward from the three “enabling skills” and downward from situation awareness. As a 
result, the movement through the three “essential skills” is from top to bottom. All of 
this supports the contention that flight examiners go about assessment conceptually, 
first in terms of building a big picture of the performance and then attributing scores 
according to the big picture and following the hierarchical order presented in the formal 
assessment model. 

Cognitively seen, Process B describes a creative event in which the flight examiners 
construe a coherent story with descriptive and explanatory events. The story has an 
overall plot, and pilots are protagonists in the plot. This places particular constraints 
from the perspective of the narrative, which needs to “hang together” to be plausible and 
convincing. This is certainly the underpinning of the sense that good flight examiners 
are good storytellers (see Chapter 2). The “writing” of the story evolves over the course 
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of an examination. In the course of the evolving descriptive statements, explanatory 
statements also enter the unfolding composition. The explanatory statements are part 
of the coherent framework that flight examiners produce (see Chapter 2). The concern 
for the story takes precedence over the assessment of performance along particular 
performance components that make up the assessment model. 


OVERALL AND CATEGORY ASSESSMENTS VARY WIDELY 


Existing research not only shows that examiners differ considerably in their assessments 
of flight performances but also that the inter-rater reliability measures continue to fall 
considerably short of those values that are acceptable within the field of psychology, 
cognitive science, or learning sciences (e.g. inter-rater reliability > .85). In these fields, if 
researchers do not achieve sufficient agreement in the coding of their data, the resulting 
measurement outcomes are considered untrustworthy. This, of course, also is of concern 
within the airline industry, which aims at assessments that are consistent from flight 
examiner to flight examiner. In the present research, participants included nine pairs 
of pilots at the ranks of first officer (n = 3), captain (n = 3), and flight examiner (n = 3) 
from each of two airlines. One of the airlines used a formal model of pilot performance 
(Airline A, see below) whereas the other one did not (Airline B). Each of these 18 pairs 
rated the individual performances of the crewmembers in the three scenarios described 
above. The results exhibit considerable variation but also some consistent patterns 
(Weber et al. 2016). First, across all three scenarios, the overall ratings of the raters 
from Airline B (broken line) scored crewmembers considerably lower than the raters 
from Airline A (solid line), where a combined score was computed from the ratings of 
the six explicit factors (Figure 3.2). 

Second, the relative scores for two crewmembers in the three scenarios are 
consistent across the two airlines in all but one case. Thus, in both the Missed Approach 
and Limited Fuel scenarios, the captains’ overall scores (white circle and square) are 
lower than those of the first officers (grey circle and square); the only difference exists 
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Figure 3.2 Mean performance ratings that pilots from two airlines (circles, squares) at three levels of 
career (first officers [FO], captains [CAP], and flight examiners [FE]) attributed to the crewmembers 
(captain [Cap], first officer [FO]) in three scenarios. 
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in the higher score that the flight examiners of Airline B attributed to the captain in the 
Limited Fuel scenario (Scenario 2). In the third scenario (Engine Fire), the first officer 
and captain ratings are identical but for the first officers from Airline A (Figure 3.2). 
In both airlines, the flight examiner mean ratings are considerably lower than those at 
the other ranks. A closer look at the rating sessions reveals that the difference is related 
to one important piece of information available in the Engine Fire scenario. When the 
captain orders the evacuation of the aircraft to the right, the engine on that side still is 
running, a fact available from the Np gauges (Figure 3.1). However, only one of the nine 
rater pairs in Airline A (flight examiner) and two of the nine rater pairs from Airline B 
(both at the rank of flight examiner) perceived this fact in the videotape. This is so even 
though the engine can be heard running. Although individual facts do not generally lead 
to pass or fail decisions (but see Chapter 7), in the present case the demarcation is very 
clear and consistent: those perceiving the evacuation order to the side of the running 
engine fail the two crewmembers based on this fact, whereas those pairs who do not 
perceive this fact all attribute pass ratings to both crewmembers. 

Scoring variance frequently is attributed to the subjective nature of the scoring 
process when conducted by individuals (e.g., Horng et al. 2010). However, even when 
assessors work in pairs to arrive at a common score—forcing them to ground their 
judgment in observable facts—considerable score variation remains between rating 
pairs (Mavin et al. 2013). The following provides a closer look at the ratings that pairs 
of pilots of different rank attribute to the captains in Scenario | (Missed Approach) and 
Scenario 2 (Limited Fuel Reserves). 

Considering the assessment of the captain in Scenario | (see box in Chapter 1), 
the scores attributed by the assessing pilot pairs vary considerably between groups and 
on each category (Figure 3.3). The three panels show, for example, that scores for the 
scenario captain’s decision-making (DM) ranged from a low of | (flight examiner pair) 
to a high of 4 (two first officer pairs). Similarly, knowledge (KN) was scored from a low 
of 1 (captain pair) to a high of 4 (first officer pair). Five of the six captain pairs (flight 
examiners also are captains) rated the six categories in a way that would have led to a 
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Figure 3.3 Pilot performance ratings on six human factors that pilots of different rank (First officers, 
captains, and flight examiners) attributed to the captain in Scenario 1 (Missed Approach). The metric 
used divides the six performance categories into enabling skills—communication (COM), management 
(MN), and knowledge (KN)—and essential skills—aircraft flown within tolerances (TOL), decision- 
making (DM), and situational awareness (SA). 
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failed examination entailing retraining; the sixth pair rated only two categories at the 
2 level and, thereby, just passed the captain. As shown in Chapter 2, flight examiners’ 
method includes an overall assessment (“would I let my family travel with this pilot?”) 
and, based on this assessment, they may adjust an individual category score such that 
it leads to a pass or fail. On the other hand, all first officer pairs passed the scenario 
captain. On completion of the assessment, the researchers asked all assessors if they 
were happy with the final scores; all groups were confident in the ratings, which they 
deemed to be consistent with their overall sense that the captain should have failed or 
passed this examination. Statistical tests comparing the captain pairs (including flight 
examiners) and first officers pairs show significant rating differences on the factors 
situation awareness, decision-making, knowledge, and management; there are no 
statistically reliable differences on the two remaining categories (flying the aircraft 
within tolerances and communication). 

Consider the case of ratings pertaining to Scenario 2 (Roth et al. 2014b). The score 
variation exists for the captain flying this scenario even though the performance assessors 
repeatedly replay scenarios and parts thereof (Figure 3.4). This is in contrast with those 
situations where scoring is done without the opportunity to return and consider, such 
as it happens during normal LOFT assessments. Here, too, the results clearly exhibit 
considerable variation. In the present instance, score variation is especially notable in 
the knowledge (KN) component, where the performance ratings ranged from a low 
of 1 (unsatisfactory) attributed by one of the captain pairs to a high of 5 (very good) 
used by one of the first officer pairs (Figure 3.4). Out of the nine assessments of the 
decision-making component, four scores were 2 (minimum standard). The knowledge 
(KN), management (MAN), and communication (COM) components also had two 
assessments at minimum standard (score = 2) rating. 

The results show that two of the captain pairs considered the captain’s performance 
as 1 ona single category, and, therefore, as an immediate fail; and two further groups, 
with two “minimum” ratings each, came close to recommending a repeat assessment 
(Figure 3.4). Thus, for example, flight examiners suggested being happy with the 
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Figure 3.4 Pilot performance ratings on six human factors that pilots of different rank (First officers, 
captains, and flight examiners) attributed to the captain in Scenario 2 (Limited Fuel Resources). The 
metric used divides the six performance categories into enabling skills—communication (COM), 
management (MN), and knowledge (KN)—and essential skills—aircraft flown within tolerances (TOL), 
decision-making (DM), and situational awareness (SA). 
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fact that both pilots passed the examination in the scenario. But examiners also noted 
deficiencies that required being dealt with, especially in the case of the captain. At the 
end of the assessment exercise, all assessor pairs indicated feeling comfortable with 
their performance assessment, reasons provided, and with the pass/fail recommendation 
that they have made. 

In this research, each assessor pair rated a total of six scenario crewmembers 
(three captains, three first officers) on six performance categories. One measure 
of rater agreement is the (Pearson’s) correlation coefficient (r). If this coefficient 
is computed, then the following results are obtained: for one of 36 correlations, 
r > .7; for four pairs, 0.6 <r < .7; and for three pairs, 0.5 < r < .6. All other values are 
r<.5. Cohen’s kappa (x) provides a better estimate of inter-rater reliability, because it 
accounts for the possibility of chance agreements not taken into account by Pearson’s 
r. When kappa is calculated, only one combination of rater pairs attains strength of 
agreement at the level of fair (« = .34), whereas all other kappa values are at the level 
of poor (x < .20). One obvious drawback underlying these considerations is the fact 
that the instrument was never scientifically validated and calibrated. On the other hand, 
the airline, in cooperation with its consultant, not only developed the instrument but 
also conducted rater alignment sessions in its CRM courses (Mavin et al. 2013). This 
should have improved the situation, for consistency can be achieved even when the 
categories themselves are not used in a scientifically correct way. Pilots not only rated 
the performances of scenario pilots but also discussed the ratings among each other 
and with the CRM course leaders (including the consultant). As noted above, there is 
evidence that training sessions where pairs or groups code a number of samples jointly 
tend to increase inter-rater reliability, which improves further with the total number of 
jointly coded episodes (e.g., Schoenfeld 1992). Moreover, given that the performance 
assessment instrument is grounded in a folk model of performance, pilots are familiar 
with the concepts used; and in the present situation, this familiarity is increased by the 
fact that all pilots have been evaluated with this same instrument. Thus, rather than 
finding an increasing alignment in the assessment scores, the airline in question finds 
itself in a situation where its work toward greater consistency in performance rating is 
thwarted. In a somewhat derogatory sounding manner, critics may gloss these results 
by saying something like: “Everyone is using the same (folk) concepts, but are doing 
so without recognizing that they use the concepts differently”. This gloss reinforces 
those critics who classify situation awareness and other concepts to be parts of folk 
models (theories). The work described here was specifically designed to investigate and 
discover the reasons for the lack of consistency. The remainder of this chapter shows 
that there are actually good reasons for the variations. Given that the assessors worked 
in pair there is further assurance that all reasons used to justify the assessment of the 
individual performance categories are intelligible, plausible, and consistent. These 
reasons are recognizable, intelligible, plausible, and reasonable within the culture 
of flight examiners; and they are such even if there are disagreements within pairs, a 
situation virtually not observed in any of the 18 pairs from the two airlines. 

Research tends to attribute lack of rater agreement to error variance. In such 
attribution, the assumption is made that there are measurement variations of unknown 
source, much like when repeated measurements of the length of an object by the same 
or multiple individuals lead to varying results. However, observing the pilots while 
doing the assessments and articulating their arguments has to lead any observer rather 
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quickly to the hunch that score variations occur in the face of good reasons. In fact, the 
findings described in the sections that follow have their origin in the observation that 
the pilots were going about their assessment exercises in a very rational way but still 
arrived at very different scores. Even a most cursory analysis of the reasons provided 
for assessment reveals that the assessors isolated specific facts that were not consistent 
across the pairs. For example, in the analysis of Scenario 2, some assessors note that 
the first officer who is pilot flying in the scenario does not confirm the setting of the 
missed approach altitude and the captain fails to seek this confirmation. In a similar vein, 
assessors note that the missed altitude capture mode is not mentioned in the cockpit, 
which it should have if the pilots were following company standard operating procedures. 
Some assessors also note that the captain in the scenario pulls the checklist but then 
puts it back and does the checklist by memory. These same facts do not become topic 
in other assessor groups, where, however, other facts are perceived and articulated. For 
example, one group notes that the first officer does not verbalize the fact that a warning 
signal shows up on the advisory display unit (ADU), but that he actually addresses the 
issue by changing the setting on the rudder trim. One assessor group notes that there 
is a discrepancy between the flight level between what the first officer asks for (flight 
level FL170) and the one that the captain (monitoring, non-flying pilot) requests from 
the tower (FL150). 


VARIATION IN NATURE AND FREQUENCY OF FACTS 


The ratio of discrimination and consistency are two critical components that distinguish 
experts from non-experts better than any other measure (experience, certification, 
social acclamation, consistency and consensus reliability, discrimination ability, or 
knowledge). Discriminating relevant information or processes therefore is integral 
to being a good judge of performance. As one study among doctors shows, although 
some cues are more important than others, there is a great variability in both diagnoses 
and cue utilization when symptoms of heart failure are present (Skaner et al. 1998). 
Existing studies of expertise suggest that there may exist correlations between level of 
expertise and problem format and the assessment of performance outcomes. Thus, for 
example, the level of expertise and the particulars of case (e.g., the type of chemical 
representation) mitigate the performance of the assessor, such as the number and types 
of categories created, the particular nature of the aspects that figured into the classes 
generated, and the function of these aspects during the process of classification. In the 
present study, for each of the six main categories, there is considerable variation with 
respect to (a) the nature of the fact used as evidence to support an assessment, (b) the 
number of facts used as pieces of evidence, and (c) the gravity with which a fact counted 
against the pilot. Consider some of the variations observable with respect to the events 
within a few seconds of the announcement of the Missed Approach scenario. This part 
is critical to all evaluations, leading some assessor pairs to fail others to pass the pilot 
flying (captain). It is critical specifically to the assessment of the situation awareness 
category, which three groups score at a 2 (poor) level, and one group ata 1 (very poor) 
level (Figure 3.3). The analysis shows variations with respect to the number of facts 
that assessor pairs extract for the purpose of assessment and in the number of rater pairs 
that discuss a particular fact (Table 3.2). 
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Table 3.2 Some of the Facts Articulated as Part of the Reasons for Assessment. 





















































First Officers Captains Flight Examiners | Sum 

A/P taken out ++ ++ ses ae qe 5 

Didn't cancel warning Sid 1 

A/P going on and off ++ | ++ | ++ ++ 4 
Go-around call (missed) ++ | ++ 2 
Go-around button ++ | ++ | ++ ++ | ++ | ++ | + | + 8 
ADU 

Low bank, IAS, heading ++ ++ | ++ ++ | ++ | ++ 6 

High bank, VS down ++ ++ 2 

Not set ai 1 
Bars 

Flying above (correct) Ela tF E 3 

Flying above (incorrect) pan 1 

Heading bar ++ died 2 
Pitch tet ++ 2 
Sum 3 2 4 2 5 4 7 5 5 






































The table shows that the nine pairs discuss between two and seven ofthe 12 facts; 
considering the frequency with which specific facts are discussed, the table shows that 
between one and eight groups note and incorporate a specific fact in their considerations. 
The featured variations are representative of the more than 60 facts one or more groups 
noted in the scenario as a whole. In the course of their assessments, all pairs tended 
to agree that the pilots have some difficulties with the go-around procedure, that the 
flying pilot (captain) displayed confusion, minor errors, or lack of familiarity (as per 
performance descriptions from the assessment form). There was considerable variation 
in how specific facts influenced the final score. For example, in the knowledge category, 
there is considerable overlap in the seven and eight pieces of evidence used by two 
pairs. One of these pairs concludes that the performance should be rated as an automatic 
fail (score = 1, very poor), whereas the other pair attributes to the captain the highest 
score (4, good) for similar facts. 

To test whether there were any surface-level differences between pairs who failed 
the pilot in the scenario versus pairs who arrived at a pass, the number of specific facts 
articulated by each pair is considered. The pairs rating the performance as a fail discussed 
19 facts, whereas the four groups rating the captain’s performance as a pass, 13 facts 
are discussed. This difference is statistically reliable (Roth and Mavin 2015). There is a 
clear difference between the two groups that pass versus fail the captain. In the groups 
assessing the performance as a fail, one flight examiner pair stands out—different in 
process and number of facts considered, more generic, and had to be encouraged to 
complete the assessment instrument. Among those passing the scenario pilots, one first 
officer pair stands out by the number of facts considered: They highlighted 18 facts. 

An investigation of the facts articulated in the course of justifying a particular 
assessment score shows that across the scenario as a whole, only one fact is articulated 
by all groups: the fact that the PF is uncertain about the turn direction upon going IMC. 
Eight groups either see that the captain fails to push the go-around button or infer 
the failure to do so from the instrumentation, where the flight director remains in its 
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original configuration telling the pilot to turn left and go down (see Figure 1.5)—thereby 
contradicting the actual behavior of increasing the plane’s attitude for returning to the 
missed approach altitude and towards the required heading (Table 3.2). Only two groups 
stated that the go-around call actually is missing (see Scenario 1.2), which would have 
been the obligatory first step to be taken according to the standard operating procedures. 


HOW FACTS INTEGRATE INTO THE ASSESSMENT 


How pilots assess the performance of peers is a function of what they observe to be 
happening, that is, their objective sense. Thus, for example, it is apparent that if a 
particular demonstrable fact is not perceived it is not going to enter the assessment. 
Chapter 2 shows that there is a constitutive relation between the objective, fact-based 
sense and the documentary sense that is derived from and explains the objective sense. 
In the context of the present work, we are concerned with the documentary sense related 
to the cultural human factors concepts. Thus, the sense of pilot performance with 
respect to a factor such as situation awareness (SA) derives from the set of SA-related 
observations O; that are taken to be documentary evidence in one or another way. We 
may write this more formally as SA = (Of, 0$,...05). Once considered in this way, it is 
immediately apparent that the more observations exist the smaller will be the influence 
of one or another individual observation on the overall documentary sense. But this 
does not have to be, as seen in the case of Scenario 3, Engine Fire Before Landing. 
As pointed out above, assessors who noted that the captain commanded an emergency 
evacuation on the side of a still-running engine (as per the Np gauge, Figure 3.1) rated 
the performance as failed whereas those assessors who did not notice this fact gave a 
pass. That is, when a fact is noted, it may play a differential role in the determination the 
documentary sense for one or more of the human factors assessed. The study of pilots 
assessing pilot performance shows that the same segment from the scenario (raw data) 
may become evidence for a different human factor responsible for the performance, 
which in part is the result of the opposite process, whereby the documentary sense 
determines how reality is perceived. 

In the investigation related to Scenario 3.1 (Missed Approach, see box Chapter 1), 
two issues turn out to weigh on the assessment. First, when the aircraft is approximately 
level with the end of the runway (Figure 1.1), the captain notes a rain cloud in the area 
where the circling is to occur but apparently does nothing to anticipate the event (e.g., 
avoid the cloud or prepare for a go-around procedure). Second, when the aircraft enters 
the raincloud, the crew appears startled, there is uncertainty about the direction of the 
turn following the missed approach call, and the actions do not follow those prescribed 
by the standard operating procedures of the pilots’ airline. 

The first segment to be considered relates to the statement that the scenario PF 
makes concerning the weather. After the comment “the rest doesn’t look to crash-hot 
out to the east there, we’ll have to be careful”, which the PM acknowledges, the crew 
does not further talk about the weather and simply carries on. For some viewers, the 
segment manifests a lack of situation awareness (see Chapter 1). Other viewers consider 
the segment to manifest poor decision-making. Still other viewers do not particularly 
attend to the comment at all; and it therefore does not influence the ways in which 
they assess the PF. Those viewers who do note the making of the statement tend to 
weigh it heavily against the PF. They do so because not acting on the bad weather 
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constitutes a major threat to the safety of the aircraft. Whether the fact is part of the 
objective sense that documents decision-making or whether it is part of the objective 
sense that documents situation awareness, the score attributed to the respective human 
factor is the lowest among the six scores the assessors attribute. For example, a first 
officer pair assigning a score of DM = 2 to decision-making (DM), a captain pair rating 
situation awareness as a 1, and a flight examiner group attributing a score of DM = 1 
(Figure 3.3) all do so based on the observation that the PF notes bad weather without 
acting on it. How do examiners use the observation to make their cases for situation 
awareness or decision-making? 

The assessment instrument subdivides the situation awareness category 
(Table 3.1)—in this being consistent with the scientific definition of situation awareness 
(Endsley 1994b)—to include perception, comprehension, and projection. When 
assessors consider this part of Scenario | in terms of situation awareness, they do so 
based on the pilot’s perception of bad weather and the lack of projection of this fact with 
respect to the future flight path. Having considered these two components, a possible lack 
of situation awareness is then used to view the video segment as manifesting a lack of 
comprehension. In this line of reasoning, a lack of understanding and comprehension is 
further supported by the fact that the two pilots appear startled when they enter the cloud 
upon engaging in the final turn (i.e., circling maneuver). That is, between noticing the 
weather and the missed approach, the crew in the scenario is not considering and making 
arrangements for a possible go-around procedure. This is part of a form of reasoning 
where a (weather-related) situation is perceived but not really comprehended; this is 
understood as the origin of the failure to project into the future. Projection actually is not 
completely absent; at least, right after stating that the weather is bad, the PF anticipates 
the need to be careful (see also Scenario excerpt 4.1, box in Chapter 4). That is, there 
is a projection noticeable at that instant, which also means that there is a certain level 
of comprehension. But this initial projection is not carried further. 

In this form of reasoning, the construction of a story from pieces of evidence is 
apparent. Any missing pieces are filled in such that the overall story line coheres. In 
this particular instance, there are two observations, one related to the perception of bad 
weather and the other about the startled pilots when they enter the bad weather earlier 
perceived. The story becomes coherent when the narrator inserts into the plot the PF’s 
lack of understanding. In that story, not all parts of situation awareness are absent or 
problematic, for the pilots do perceive what is happening outside the aircraft. But 
the other two components are taken to be absent based on the fact that at the point of 
entering the cloud, its presence is an apparent surprise. 

The preceding analysis shows how assessors may consider this aspect of the flight 
as evidence for problems with situation awareness. When coming to the rating part of 
the assessment task, viewers ultimately may arrive at the sense that situation awareness 
should be rated 2 (poor). On the one hand, the PF has “perceived significant systems 
or environmental factors affecting flight” (SA1.3). On the other hand, the PF notes 
that care has to be taken and, therefore manifests some form of comprehension and 
projection, which makes the descriptions “did not comprehend” (SA2.1) and “did not 
predict” (SA3.1) not entirely appropriate. At the same time, the PF apparently does 
not “comprehend significant factors” (SA2.3) and has more than “some difficulty 
predicting future events” (SA3.3). As a result, assessors may conclude that a rating 
SA =2 is the most appropriate. The PF in the scenario realizes what is happening outside 
the cockpit but then does not project the implications of the outside situation to the 
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inside. Other observers, however, may weigh the observation that there is recognition of 
bad weather and at least initial understanding-presupposing projection as documentary 
evidence for situation awareness that is “not that bad”, which is used in support of an 
overall rating of SA = 3. 

The same instant from the scenario may also be part of making the case for poor 
decision-making. This may actually reflect the situation of the assessment made in the 
case of the Lao Air flight QV 301, which involved perceptual issues and wrong climb 
directions, but which is attributed to a problem of decision-making because the crew 
continued the approach to Pakse airport despite the sudden change of weather condition 
(see box, Chapter 1). In the present research, viewers making a case for problematic 
decision-making may note that the scenario PF sees poor weather conditions but does 
nothing about it. When the instant is seen from a decision-making perspective—i.e., as 
documentary evidence of (poor) decision-making—the lack of option generation comes 
to be noted (see Figure 4.4 below). Alternative options include descending below the 
cloud to avoid it or debrief a plan for conducting a missed approach in the case that the 
aircraft does indeed go IMC. As a result, there is only one (reasonable) option when the 
aircraft enters IMC. Option generation and contingency planning both are subcategories 
of decision-making. As part of such a story centering on decision-making, observers 
may note that there is no evidence for diagnosis of the situation. As a result, the crew 
does not consider the risk that arises from entering IMC upon making the final turn. 
Moreover, once the documentary sense of poor decision-making has become the initial 
fix, assessors may begin attending to the timing. That is, they may note that diagnosis 
and option generation should begin as soon as the crew recognizes bad weather so that 
the absence of evidence for immediate (i.e., timely) diagnostic and option-generating 
actions are used as documents of poor decision-making. As a result of noting an 
absence of evidence for timeliness, diagnosis, option generation, risk assessment, 
and contingency planning, assessors may conclude that poor management is the root 
cause of the problems attributed to the pilot flying. Unsurprisingly, once the complete 
absence of behaviors documenting these subcategories of decision-making (DM) is 
noted, assessors easily conclude that the category as a whole should be rated DM = 1. 

The preceding paragraphs describe and analyze the work of assessors of one 
fragment of the scenario on which much of the overall documentary sense ofa fail rating 
is based. The analysis shows how the same behavior becomes part of different story 
lines, one grounded in the sense of poor situation awareness, and the other grounded 
in the sense of poor and even absent decision-making (when there is only one option, 
no decision has to be made). A second, brief segment that plays a decisive role in the 
assessment of the pilot concerns the events immediately following the (unannounced) 
beginning of the go-around procedure (box, Chapter 1). The critical aspect of this 
part of the scenario is the statement, “Missed approach is right hand, wasn’t it?” 
Following a corrective statement, “negative, left, left”, the PF says with the same rising 
intonation as before, which affords the statement to be heard as querying, “It is left?” 
(see Scenario 1.2). 

The two statements, although recorded and heard objectively, function very 
differently in the appreciations of different evaluator pairs. We may understand this 
difference in terms of different aspects (Wittgenstein 1953/1997). Every reader will 
have had the experience with optical puzzles where one or another figure is seen—such 
as the one seen as a vase or as two faces (Chapter 2). The material drawing is the same 
in both cases. This is what allows someone to switch from one to another view. A 
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cognitive analysis of optical puzzles shows that at the bottom of the phenomenon are 
shifts in the primary focus and associated shifts in the relative weight and salience of 
parts of a drawing to the whole (Roth 2012). Individuals with a little understanding of 
the phenomenon and some familiarity with it can easily change the conditions to move 
from seeing one or the other perception. 

As aresult of perceiving different types of events, assessor come to very different 
conclusions about how to classify and rate the performance. The perception of what 
is occurring depends on the overall story line, the overall sense of the event, which 
then determines the objective sense. In turn, the objective sense becomes document 
of, and reifies, the documentary sense. Consider the following ways in which different 
assessors approach the scenario fragment under consideration. Some see the aircraft 
turning right, rolling the wrong way, and note that the PF initially is going to turn the 
wrong way or wanting to go right (sometimes without actually stating that the aircraft 
is rolling to the right). Other observers see confusion and hear the captain asking for or 
confirming the correct direction rather than just going the wrong way. Assessors who 
perceive the aircraft roll to the right or perceive the captain wanting to go right prior to 
being corrected heavily weigh this against the captain. That is, the objective sense has 
a lot of weight in determining the performance as problematic (see Chapter 4 below, 
which discusses the role of weight on determining the assessment score). On the other 
hand, when the captain is perceived as seeking confirmation about the direction in 
which to fly the missed approach, then this counts heavily in favor of the captain and 
the positive climate on the flight deck. The fact supports a story line whereby the PF 
is confused, but deals with the confusion by communicating with the PM. This allows 
lack of knowledge and confusion to be acknowledged, brought out into the open, and 
to be corrected prior to engaging any procedure. In such a story plot, the resulting 
correction may be perceived as error trapping, which is then used in favor of the PM, 
who, by asking or confirming the direction, initiates the process. 

In evaluating this flight segment, many observers do not attempt to ascertain 
the actual aircraft movements based on the evidence that can be extracted from the 
instrument (which is not always seen). But some viewers agree that there is evidence in 
the video that the aircraft is turning to the right—the banking angle is about 3 degrees 
to the right—before being corrected and then turning to the left (see Figure 1.5). This 
observation may be used as objective evidence for the sense that the aircraft is flown 
beyond tolerance, and thus becomes documentary evidence against the pilot on the 
assessment category Aircraft flown within tolerances. 

On the one hand, the scenario provides assessors with objective evidence for an 
event in the sense that what the scenario pilots have said and what the aircraft has done is 
available for everyone to see, hear, and evaluate. Yet what the assessing experts actually 
take as facts from this data source differs, that is, there is a variation in the nature of the 
actual data that they use in support of their assessment. Indeed, the objective sense is 
determined by the overall story, based on a documentary sense of the pilot as a whole, 
which in turn is based on the objective sense. As shown in Chapter 2, the understanding 
that manifests itself in the scenario descriptions entails explanation, and explanation 
develops understanding descriptions. Thus, depending on the initial unfolding sketch 
for an overall story, a specific micro-event may be perceived in terms of one or another 
human factor that dominates the explication. Observations and explanations then reify 
one another, leading to an overall assessment associated with a specific story line. 
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Unsurprisingly, from this perspective and repeating a comment made above, good flight 
examiners are said to be good storytellers—though it actually may be the other way 
around: because someone is a good storyteller peers may be led to attribute the person 
the status of a good flight examiner. 


STORIES AND THE PREVALENCE OF SYMBOLIZING GESTURES 


When people narrate stories and describe situations, they tend to produce spontaneous, 
idiosyncratic gestures (gesticulations) together with speech (McNeill 2005). This is so 
independent of culture and of the actual presence of the interlocutor (e.g., while talking 
on the telephone). Even blind people gesture when talking to others. These gestures 
are co-produced with speech without conscious awareness; they come in the form of 
beat, deictic (i.e., pointing), and iconic (metaphoric) gestures. There are differences 
in the kind of gestures used according to the situation. When speakers are oriented 
towards diagrams and photographs, deictic gestures and two-dimensional iconic 
gestures following features in the representation are prevalent; and when speakers 
are oriented towards the audience, with or without representations present, gestures 
are associated with a three-dimensional “narrative space” (Roth and Lawless 2002). 
One may therefore anticipate that there will be more gestures relevant to the flight in 
those groups that constructed (longer) narratives that are mapped onto the assessment 
metric rather than those groups that are focused on the assessment instrument as the 
organizing feature of the task. 

There are three types of iconic gestures notable in assessment situations: (a) 
reenacting movements specific to the operation of the aircraft, such as pushing buttons 
(e.g., go-around button), turning knobs, and moving control column; (b) featuring 
the orientation or flight path from the perspective of the pilots in the cockpit; and (c) 
modeling a flight path as seen from the outside. In the process of talking about what 
should be done to initiate the go-around procedure, assessors often move their fingers as 
if pushing the go-around button (Figure 3.5a), which is followed by moving the power- 
lever forward. Assessors use iconic gestures that indicate the nature of the instrument 
read (e.g., what they would see on the flight director, where the bars were, and how 
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thing, when going The command bar to circle left. 
around. was down, still down On two occasions, 
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off into circling... 
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Figure 3.5 Symbolizing gestures that are part of the assessment stories. (a) Pushing the go-around 
button. (b) Flight path from the perspective of pilots. (c) Bank angle and pitch during a left-oriented 
go-around. (d) Third-person perspective on the orientation of an aircraft. 
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they are oriented). A downward gesture may depict what pilots would see on the flight 
director, where the horizon “would be way down low”, incorrectly telling the pilot to 
head downwards (Figure 3.5b). In the assessment of the Missed Approach scenario, 
many gestures are related to the beginning of the go-around procedure, showing either 
flight direction or the bank angle of the aircraft in the correct procedure (to the left, 
Figure 3.5c) or incorrect procedure (when flown to the right). Thus, a statement about 
the manipulation of the aircraft may be accompanied by iconic gestures representing 
a third-person perspective on the flight, gesturing aspects of the flight path (profile) 
when seen from the outside (Figure 3.5d). Such third-person perspectives are integral 
to the overall storyline, for they contribute explanatory aspects that are important to 
developing the understanding of pilots when they are debriefed about critical events 
(see Chapter 7). 

As part of the situation description (story line or plot), assessors may indeed act out 
the scenario as if they were sitting in the cockpit, with iconic gestures directed towards 
the locations where the switches and buttons are located. The gesture is pushing, if 
the corresponding button needed to be pushed, or is a flicking gesture, if switch has to 
be thrown. We can see this when a flight examiner talks about the situation whereby 
the missing push of the go-around-button leads to the flight director telling the pilot 
in the scenario to fly down when in fact he was to raise the nose of the aircraft and fly 
up (see Figure 1.5). The examiner then says what he has quite frequently seen when 
pilots were following the flight director that pointed down (Figure 3.6a). The account 
is accompanied by a gesture that is holding the control column (Figure 3.6b), which is 
then pushed forward (Figure 3.6c). This would have made the aircraft head downwards. 

The result of these actions manifests itself in gestures symbolizing how the aircraft 
would fly from the perspective of the flight deck: an approximately parabolic flight 
path that comes to an end with a bump (Figure 3.6d). Talking about the scenario, flight 
examiners may explicate that it is dangerous to leave the command bars pointing down 
while again gesturing the position of what the pilot would have seen, his actual attitude 
and the position of the flight director bar way below it. Leaving the flight director bars 
pointing down is especially dangerous (Figure 3.6e) when the aircraft is close to the 
ground, a statement that was accompanied by a gesture that symbolizes a changing 
banking angle as the aircraft makes its final turn to the left to align itself with the runway 
(Figure 3.6f). The flight examiner therefore becomes the center of the story, sometimes 
acting out what the pilot does, and sometimes what the pilot and aircraft experience. 
The plot and actor come to live in the performance of the flight examiner. The result is 
a little drama, representing the drama of life. 

Striking differences existed in the use of gestures associated with the assessment 
process. Thus, the flight examiners, whose assessments correspond to Process B, 
employed significantly more iconic gestures in the course of assessing the captain’s 
performance in the scenario (about 82 gestures) than the captains and first officers who 
employed Process A (about 23 gestures). One might assume that the number of gestures 
is related to the amount of time talked, the number of facts talked about, or the number 
of times the assessor pairs actually watched the videotape (intensity of engagement). 
However, the number of iconic gestures employed was not significantly correlated with 
the amount of time spent on the assessment, the number of flight-related facts stated, 
or the number of times the pairs returned to the video to watch particular segments. 
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The flight director suddenly their brain oh there's the flight 
indicating fly down... explodes ... director heading 
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Figure 3.6 Series of symbolizing gestures part of a story in which following the false directions of a 
flight director indicator when the go-around button is not pushed. The gestures symbolize: (a) flight 
director bars as seen by the pilot flying; (b) the current position of the control column; (c) the pilot 
moving the control column when following the flight director bar; (d) third-person view of the aircraft 
plunging to the ground; (e) the flight director bars viewed by the pilot flying; and (f) the downward 
left-rolling movement of the aircraft had the pilot followed the flight director bar. 


The prevalence of iconic gestures re-enacting the movements required to fly the 
aircraft and making salient where and what to read from the instrument panel, associated 
with the gestures about the flight path of the plane from the perspective of the pilots 
is in support of the kinetic approach to knowledge of flying (see Chapter 1). Thus, 
assessors often move through an entire kinetic melody of hand and eye movements 
required during a go-around procedure in the way it should be flown according to 
the standard operating procedure (see Figure 1.7). Indeed, in reconstituting a correct 
procedure, pilots do not construct a narrative based on individual facts recalled from 
memory. Instead, they initiate the production of an account by symbolizing the thumb 
movement pushing the go-around button followed by the movement required to push 
the power levers to the go-around position at the beginning of the ramp (torque TQ up 
to 100%). The use of iconic (symbolizing) gestures also fits together with the flight 
examiners’ efforts to produce complete narrative plots that pull together facts and 
explanations. These plots are based on event-like evidence rather than on individual 
facts, even though the latter are integral parts of the evolving documentary sense for 
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each of the performance constructs. Further evidence for the holistic nature of the flight 
examiners’ method exists in their gestural sequences that pull together the coordinated 
kinetic melodies of pilot flying and pilot monitoring into a single hole. Thus, consistent 
with the holistic view of the joint cognitive system, they act out the entire go-around 
procedure in one sequential and coordinated movement: pushing the go-around button, 
moving the power lever to the ramp, and pulling the control column (PF), followed by 
the PM’s selection of the flaps, selection of gear up, and movement to the automatic 
flight control system control panel where the heading mode, bank angle, and speed are 
set (Table 1.1). Rather than having two separate but coordinated movements, flight 
examiners integrate the two movements into one single melody (Figure 1.7). This 
corresponds to how they perceive events during an examination, where they do not 
separate out individual components but observe the events as wholes, which after the 
fact may be translated and decomposed in the narrative in the transition from mimesis, 
to mimesis,. That is, they move from the transactional world actually witnessed to a 
represented world where there are individual actors with definite skills and who interact 
with other individual actors. 


FROM MEASUREMENT ERROR TO REASONED CATEGORIZATION 


From all of the evidence mobilized in this chapter emerges a picture of the complexity 
of assessment, which turns out to be a process of establishing a coherent account 
(narrative, story) that makes sense of and accurately—from the perspective of the 
narrators—represents what can be seen in the videotaped scenario. In this picture, we 
do not see arbitrary variations constituting measurement error treated as the root cause 
of score variation and statistical variance when performance assessment is treated 
from a measurement paradigm. There are differences in the pass/fail ratings and in 
the associated reasons provided in justifying the rating, along the lines of having done 
this kind of work for a while. There are also differences with respect to the overall 
process, whereby the most seasoned flight examiners constructed holistic narratives 
of the situations to be evaluated, drawing, to a significantly larger extent than their 
peers, on bodily forms of aviation knowledge (i.e., gestures, body orientations). The 
detail provided by the assessment tool, for this type of assessor, was less important 
than the overall narratives they were producing together for the purpose of capturing 
the essence of the episodes. This is consistent with flight examiners stating that they 
generally remember the big picture rather than the specific facts. On the other hand, 
the assessment tool organized the assessment processes of captains and first officers. 
These pilots first selected categories and elements and then looked for evidence in 
support to matched the word pictures (behavioral descriptions). The process was less 
about establishing some integrating, holistic picture and more about accumulating facts 
required to rate the individual assessment items. Previously conducted research suggests 
that performance assessment is constituted by a sequence whereby identified facts are 
attributed to categories, which are then analyzed element by element (O’Connor et al. 
2002). On the base of the implicit or explicit combining rules, an overall assessment 
is obtained. In the present work, a more differentiated picture emerges. In the case of 
the less experienced experts, the process can therefore be represented in the structure: 
categories > elements > facts/stimuli > (implicit, given) combining rules pass/fail. 
Here, the combining rules tended to be given—company rules specified that three or 
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more 2 (poor) ratings or one 1 (very poor) rating means failing the examination—or, 
within a category, implicit because pairs tended to go with the score that combined 
the majority of elements (word pictures). In the case of the flight examiners who were 
more seasoned as assessors, a different picture emerges: holistic description > (implicit, 
explicit) pass/fail > categories > elements. 

There were differences in the way specific facts and specific events from the 
episode entered the assessment process. Previous studies show that as individuals 
develop greater expertise, they discriminate larger and more relevant amounts of 
information during situational appraisal (Weiss and Shanteau 2003). In regard to those 
pairs who failed the scenario pilots, there was a significant increase in the number of 
facts/evidence used to determine the fail grade. By simply totaling the number of facts 
identified would clearly determine the pass/fail decision. However, further analysis 
yields a far more complex and differentiated scenario. For instance, once a fact had 
been identified—or in some cases not even considered—assessor pairs categorized the 
same facts differently (e.g., situation awareness versus decision-making). Further more, 
the weighting that occurred for each fact was found to differ. This was apparent across 
all pairs. The picture is complicated by the structure of the assessment process. As 
part of constructing the overall, more-or-less coherent, intelligible narrative, the more 
seasoned pilots (flight examiners) identified salient facts/stimuli. These tended to be 
associated with a key aspect in the flight—e.g., noticing the bad weather and what was 
or was not done subsequently or the issue of querying the turn direction upon the missed 
approach actions. This, again, confirms interviews with flight examiners who tend to 
identify a key instance and associated performance aspect that leads to failing a pilot. 
In the assessment tool-driven process, facts were identified to dis/confirm the current 
behavioral description (element). However, in both types of approaches, the assessor 
pairs subsequently determined that they were confident that their ultimate result, pass 
or fail rating, corresponded to their overall sense of the quality of the performance. 
That is, at no point was the sense detected that the company rules for combining the 
categories into an overall assessment was forcing a decision other than that based on 
a general sense of how well the assessed pilots actually did. 

From a practical perspective, the present findings raise questions about inter- 
rater reliability generally and about increasing reliability of on-the-job performance 
assessment specifically, especially because the outcomes can have tremendous effects on 
the lives of the pilots (whether they are promoted or lose their jobs). Concerning inter- 
rater reliability, the present chapter shows, for example, that the same micro-event—the 
flying pilot’s (captain) utterance “missed approach is right” with rising intonation—can 
be perceived very differently: as a query seeking confirmation or simply as an indication 
of lack of knowledge and procedures. In the former way, the utterance can be assessed 
as a positive aspect: good communication (one category) or as poor situation awareness 
(another category). In the present instance, the PF produces the utterance (question). 
An additional level of complexity would enter the assessment if the PM (first officer) 
had been the flying pilot because of the power-over relationships that continue to exist 
in the cockpits of many companies and pilot pairs. It may be uncertain whether the two 
most commonly used approaches to assessment—behavioral observation training and 
frame of reference training (Woehr and Huffcutt 1994)—could improve the reliability 
of performance assessment. 

Behavioral observation training focuses on improving how assessors observe and 
retain evidence during the assessment process. With respect to the latter, the issue of 
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access to evidence was reduced because pilot pairs were in a position to return to and 
repeatedly watch the videotape or segments thereof and to their notes written while 
watching. Access to information is also made easy in the actual assessment process when 
companies use a “debriefing tool”, which essentially consists in a video that examiners 
and pilots can draw on during the debriefing session to replay specific (examiner marked) 
sequences (see Chapter 8, Figure 8.1). With respect to the identification of specific pieces 
of evidence, the present chapter exhibits the existence of considerable variation in the 
nature and frequency of the facts identified within each rater pair. Variation existed even 
though the assessors worked in pairs; and it existed even within rank. 

Frame-of-reference training focuses on aligning judgments of individuals with 
correct performance dimensions—e.g., situation awareness or communication—then 
correctly assessing the effectiveness level (weighting) of that performance dimension. 
This chapter shows that pairs identify different facts yet subsume them to the same 
category, or identify the same facts and categorize them differently. All pilots involved 
in this research have had training in the use of the assessment model and associated 
assessment tool as part of their human resource management training courses. On the 
one hand, the less-seasoned experts performed assessment processes that may appear to 
be in line with frame-of-reference training. But the frame-of-reference approach breaks 
complex situations and performances into pieces to reconstitute the overall assessment 
through implicit or explicit combination rules. On the other hand, the seasoned 
examiners—in constructing an overall narrative that reconstituted the essence of the 
overall performance—first develop a sense of whether a performance is un/acceptable 
and then map this narrative onto the dimensions of the assessment tool. That is, frame- 
of-reference training actually may lead to assessment processes that differ from those 
that are observed among the most seasoned and capable pilot-examiners. An attempt to 
changing the approach would constitute an attempt to change the culture as a whole— 
which, upon considering the stability of cultures and cultural practices, would be an 
enormous if not impossible task. Future studies might be designed to experimentally 
compare the effect of training in the construction of holistic narratives with other 
recommended methods for enhancing inter-rater reliability on the improvement in 
performance assessments. 

A second important pragmatic question concerns the nature of such an assessment 
tool given that it constitutes an integral part of the pilots’ career progress. For all pilots 
it is the tool on which their own evaluations are based; for the flight examiners, it is 
the tool that they are working with on a daily basis. In addition, some of the captains 
will be promoted to the position of flight examiner. If the complexity of the job is such 
that it makes little sense to consider performance in terms of independent performance 
elements—1.e., categories including situation awareness, management, or decision- 
making—then the precise role of the assessment instrument is at stake. Might the tool 
be used to elicit reflection on components on practice and, in so doing, contribute to 
the performance improvements as the literature on reflective practice suggests? May 
the industry benefit from considering assessment that has similarities with case-based 
law so that pilot assessment is reported not in single scores (the convenient route) but 
in collections of narratives that better characterize the performances of pilots across a 
range of situations? 
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KEY POINTS 


e Inter-rater reliability models of assessment attribute variance to errors of 


measurement. Such errors, because of their random nature, inherently become 
inaccessible to modeling. Inter-rater reliability models fail to take into account that 
assessors frequently have very good, observation-based reasons for rating pilots 
in one rather than another way. Most significantly, inter-rater reliability models 
treat assessment as a measurement issue. 


The evidence in Chapters 2 and 3 suggests that flight examiners and other pilots 
alike make qualitative, categorical judgments. These range from polar good/bad 
distinctions to three-, four-, and even five-category sets (e.g., very good, good, 
satisfactory, good, very good). Moreover, observations generally are expressed in 
inherently fuzzy linguistic categories and observation sentences; they therefore tend 
to be flexible and ambivalent. There are also fuzzy boundaries distinguishing one 
from another category, always existing in the form of (fuzzy) descriptions rather 
than as things. To model assessment in ways that account for these phenomena, 
three possible approaches are described in this chapter. 


Constraint satisfaction models take into account the fact that many observations, 
each to a different degree, tend to be taken into account to make a binary decision 
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(e.g., pass/fail). Verbal observation statements that are used as part of the assessment 
reasoning are used as information supporting the associated hypothesis. Sets 
of mutually supporting hypotheses favor one or the other of two assessment 
possibilities. The assessment outcome is modeled as the final state that the network 
reaches after it is allowed to relax. 


+ The cusp catastrophe is frequently used to mathematically model the transitions 
between two states. Catastrophe theory has been proposed for modeling the 
emergence of morphologies (e.g., the existence of a binary system) and the 
transitions between binary states when both are possible given the values of two 
control variables. Such transitions are brought about by minor (infinitesimal) 
variations in one or another of the two control variables. The dependent variable has 
two, qualitatively different states, which makes the catastrophe theoretic approach 
quite useful and applicable to assessment processes where categories are involved. 


+ The fuzzy logic approach models how fuzzy observation-based reasons transform 
to yield assessment in the face of often-overlapping, inherently vaguely defined 
and ambiguous assessment categories. The approach takes into account that the 
same observation, as shown in Chapter 3, may be indicative of different assessment 
categories. 


he assessment of pilots tends to be treated as a measurement issue together 

with the associated question of inter-rater reliability. Performance assessment 

and more specifically the assessment of pilots’ non-technical performance 
areas turns out to be difficult because researchers and practitioners find that inter-rater 
reliabilities tend to be moderate to low. Indeed, one of the airlines involved in the 
research had adopted the assessment model featured in Chapter 3 and found that there 
was considerable variation among assessors. It therefore designed its CRM courses 
around joint assessment sessions involving all its pilots in the hopes of increasing 
inter-rater reliability. On the other hand, some researchers even suggest abandoning non- 
technical skills currently in use, including situation awareness, because they correspond 
to folk psychology and have no scientific basis (Dekker and Hollnagel 2004). Chapter 
2 shows how flight examiners use the documentary method in a principled manner 
for making sense of pilot performance and for arriving at assessments together with 
an explanatory framework. Chapter 3 shows how the same micro-events or objective 
facts may become part of the objective sense that gives rise to and supports different 
performance-related human factors. Such observations appear to be consistent with 
suggestions that meaningful criteria for consistently assessing performance are elusive 
(Rigner and Dekker 2000). The observations therefore might be used in support of the 
hypothesis that assessment cannot be modeled rigorously and mathematically. However, 
Chapters 2 and 3 already articulate some reasons for suspecting that the hypothesis 
finds little support. In this chapter, three mathematical approaches are presented for 
modeling assessment: constraint satisfaction networks, catastrophe theory, and fuzzy 
logic. The first two, though outlined here in relatively brief form, allow interesting 
conceptualizations of the assessment process generally and the influence of small 
variations in factual information on the ultimate assessment outcome specifically. The 
third approach, which is relatively easy to implement without requiring a considerable 
mathematical background, is developed more extensively. It provides great predictive 
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power for the actual scores raters provide once the facts used in the reasoning and their 
intensities are known—such as given in the pairwise assessment sessions featured in 
Chapter 3. 


ASSESSMENT OF NON-TECHNICAL SKILLS AND PROBLEMS WITH 
THE INTER-RATER RELIABILITY MODEL 


In its early move toward assuring safety, the airline industry focused on the assessment 
of pilots’ technical skills. But it turned out that the technical skills frequently were not 
the root causes of (fatal) aviation incidents. Instead, the so-called non-technical skills 
including situation awareness, management, and decision-making often were invoked 
to explain why an aircraft had crashed (Helmreich et al. 2004). The final reports of 
many of the incidents referred to in this book cite the lack of situation awareness; and 
there may be statements that this lack could have been mitigated through appropriate 
flight management or decision-making. Thus, for example, the pilot flying the Fokker 
F100 that crashed near Heho (Myanmar) airport on Christmas Day of 2012 (box, 
Chapter 7) did not realize that the aircraft was off the outbound and inbound courses 
specified in the published NDB approach plate and decided to continue the landing 
attempt even though the aircraft was not stabilized when passing the missed approach 
point. The pilots of the military B737 that crashed near Dubrovnik, Croatia (April 3, 
1996) decided to land even though the required second automatic direction finder was 
not on board; it was to be used as the distance measuring equipment for identifying the 
missed approach point marked by the position of a second beacon for the non-directional 
beacon (NDB) approach. On their approach, the crew was said to have lost situation 
awareness and headed 10° to the left of the actual inbound track, leading to a controlled 
flight into a mountain with such intensity that the probability of surviving was nil. All 
five crewmembers and 30 passengers died. 

To train pilots in these other, non-technical areas, crew resource management 
training (CRM) became an important dimension of airline operations. Associated with 
CRM training, schemes were developed for assessing the “soft”, non-technical human 
factors. In Europe, the NOTECHS (non-technical skills) system was developed for 
assessing pilots’ non-technical skills (Flin et al. 2003); in the Tasman region, a number 
of airlines use the Model for Assessment of Pilots’ Performance (MAPP), which 
integrates technical and non-technical performance dimensions (Mavin and Dall’ Alba 
2010). Both approaches are tied to an inter-rater reliability model of assessment. Indeed, 
some regulators link the use of non-technical skills to the possibility of measuring 
them accurately. Thus, “only where organizations have developed good levels of 
standardization and inter-rater reliability does CASA support a move towards jeopardy 
based non-technical skills training” (CASA 2011).' In tying the use of non-technical 
human factors to inter-rater reliability, the Civil Aviation Safety Authority (CASA) 
seeks to assure that whatever factors are used they can be matched consistently with 
observable behavior. If two or more raters use different human factors categories to 
explain performance or rate the same behavior differently on a measurement scale 





' The MAPP (Mavin and Dall’ Alba 2010) has not undergone formal standardization and inter-rater 
reliability assessment. When it has been used, the inter-rater reliability actually is low (Mavin et al. 
2013). 
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designed to assess the quality of situation awareness, management, or decision-making, 
then consistency and therefore reliability is not ascertained. In the context of the MAPP, 
Chapter 3 exhibits the varying types and frequencies of facts that different rater pairs 
extract from the same flight scenario (e.g., Table 3.1). Inter-rater reliability, however, is 
an important component in the drive for global standards in the aviation industry (ICAO 
2007). A contradiction therefore exists between the collective intention to improve 
the industry and the difficulties that come with attempts of assessing non-technical 
skills rigorously and consistently. In part, other studies have shown that raters differ 
significantly not only between but also within levels of experience in terms of the severity 
with which they assess certain aspect of pilot performance (e.g., Mulqueen et al. 2002). 
Efforts designed to improve inter-rater reliability provide inconsistent results, often 
exhibiting modest consistency and congruency even after a three-year training effort 
(Holt et al. 2002). That is, industry efforts in improving safety through non-technical 
skill (CRM) training appear to have reached a performance plateau (Amalberti 2001). 
How does aviation compare with the reliability of assessment in other domains? 

Research concerned with the reliability of assessments shows considerable variation 
across domains (Shanteau et al. 2002). Within-rater reliability is high in some domains, 
for example, among weather forecasters (7 = .98), livestock judges (7 = .96), or auditors 
(r= .90). But these same measures may be quite low in other areas, for example, among 
pathologists (7 = .50), clinical psychologists (r = .44), and stockbrokers (r < .40). Of 
course, within-rater reliability may be high because raters are consistent; but they 
may be consistently following incorrect rules. When the question concerns consensus, 
then the observed reliabilities are lower. However, the professions cluster together as 
they do for within-rater reliability. Thus, the highest levels of inter-rater reliability are 
observed among weather forecasters (r = .95), auditors (r = .76), and grain inspectors 
(r = .60) whereas the lowest levels of agreement have been demonstrated to exist 
between pathologists (r = .55), clinical psychologists (r = .40), and stockbrokers 
(r = .32). In aviation, inter-rater reliability levels vary by study, reporting good 
(O’Connor et al. 2002), moderate (Flin et al. 2003), and low to non-existent agreement 
(Smith et al. 2008). Whereas there sometimes are suggestions to increase agreement by 
increasing the number of raters, Chapter 3 shows that there is considerable disagreement 
between ratings even when rater pairs work together to report one assessment (score). 

In the pursuit of understanding the issues underlying the problem with inter-rater 
reliability, consider the published data related to the MAPP instrument (see Chapter 
3). Nine pairs of pilots rated the two crewmembers appearing in three scenarios on 
six human factors. This yields a total of 36 ratings. The pairwise correlations for 
the nine rater pairs range from low, near near-zero correlation to a highest value of 
r=.71 (Chapter 3). Consider two scenarios, one modeling virtually absent agreement 
(Figure 4.1a) and a moderate Pearson correlation and fair agreement (based on Cohen’s 
kappa) (Figure 4.1b). In the former case, Rating A and Rating B assign the same score 
in only six instances (out of 36), as shown by the numbers in the main diagonal of the 
matrix where the cells are shaded grey (Figure 4.1a). The variation here is distributed 
equally between the two sets, Rating A sometimes is one point higher at other times 
one point lower than Rating B. But larger distances are also displayed, for example, in 
two instances a score of 4 by Rating A obtains a score of 2 by Rating B. In one instance, 
there are three points of difference. 
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Figure 4.1 Comparison of pairs of rating across 36 instances. (a) Low correlation. (b) Highest 
correlation in the dataset. 


The second case exhibits a considerably higher agreement with a total of 19 
instances of agreement associated with a Cohen kappa measure that corresponds to 
Jair (Figure 4.1b). The matrix also shows that overall, Rating A tends to be higher than 
Rating B, with 13 counts above the diagonal and only three counts below. Existing 
studies are not unequivocal about this tendency, one showing first officers to be more 
severe in their ratings (Mavin et al. 2013), whereas another involving some of the same 
authors and the same context reports first officers giving higher, more lenient scores 
(Weber et al. 2016). Unexplained systematic differences also are reported in other 
studies of performance assessment in aviation (e.g., Holt et al. 2002). Future research 
may or may not associate such cases with career-related differences, such as when 
the ratings of flight examiners or simple captains are compared with first officers, or, 
perhaps more useful, when total number of flight hours or years on type is used as the 
distinctive parameter. The important point for the present purposes is this: The inter-rater 
reliability approach attributes the deviations from the main diagonal to measurement 
error. Because reliability is defined as 


Var(true) _ Var(true) 
Var(observed)  Var(true) + Var(error)’ 





Reliability = 


where Var(x) is the variance associated with x, higher error variance means lower 
reliability. The problem here lies in the fact that modeling assessment in this way fails 
to distinguish fuzziness and randomness of error variance. A first observation from 
the comparisons of pairs of ratings shows that randomness may exist in the case of 
low reliability (Figure 4.1a), but the variation is rather consistent in the second case 
(Figure 4.1b). Moreover, this approach does not account for the fact that raters may 
have very good reasons for assigning scores in the way they do; and when they rate in 
pairs, as the raters in Chapter 3 do, these reasons are seen to be intelligible, reasonable, 
and acceptable. 

The evidence presented in Chapter 3 suggests that there are good, albeit perhaps 
fuzzy facts and reasons for the varying ways in which assessors of pilot performance 
order their world and in how they account for its orderly nature. This leads at the end 
of Chapter 3 to the conclusion that performance assessment in aviation may not be a 
measurement issue at all but an issue of categorization. A different approach to rater 
variance would be to consider assessment as a rational endeavor of categorization, 
even when the “inputs” into the decision-making process underlying assessment are 
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inherently characterized by uncertainty and requiring local knowledge to be understood. 
Consider again the rating scenarios. When the rater pairs are observed while they discuss 
the reasons for a specific rating, these actually may be very legitimate in the context 
of the particular facts that two groups notice and that lead to the differences in their 
evolving documentary senses concerning a particular human factor. Moreover, even 
in the simplest of cases of a line drawing, some viewers may see a duck when others 
see a rabbit, or some viewers may see the image of an old lady when others see the 
image of a young lady in the same place (e.g., Gregory 1994). Perceptual differences 
are possible because every present situation is inherently vague, a state that remains 
“invariant to the clarification furnished by [further] exchanges of questions and answers” 
(Garfinkel 1967). In fact, even the most rigorously designed survey instruments contain 
categorizations, which weaken any assurances for achieving reliability and, therefore, 
the external validity of quantitative inferences based on responses (Suchman and 
Jordan 1990). 

In Chapters 2 and 3, the flight examiners’ method is described in terms of an 
increasing affirmation of a particular documentary sense with respect to pass or fail as 
much as with respect to the assessment of individual performance categories. Here the 
issue is categorization, dramatically shown in the question flight examiners ultimately 
ask themselves: “Do I want my family to fly with this or these pilots?” The approach 
is category-based even in the case where there are scores. Indeed, the scores associated 
with the MAPP are arbitrary, for the pilots and examiners compare verbal descriptions 
to the events they observe (see Table 3.1). Only once they have found an appropriate 
verbal description do assessors identify the actual score to be assigned (see box, 
Chapter 3). Descriptions, however, make qualitative and therefore categorical rather 
than quantitative distinctions. Moreover, the scale is a translation of a set of categories 
{very poor, poor, satisfactory, good, very good} (Mavin et al. 2013) or {unsatisfactory, 
minimum standard, satisfactory, good standard, very good standard} (Weber et al. 2016); 
but there is no guarantee of any homogeneity in the translation from the categories 
to the scores. Even though these categories constitute a ranking of a qualitative type, 
there is no reason to assume that these qualitative differences map, in some linear or 
non-linear way, onto a quantitative scale; and any such translation tends to miss the fact 
that the categories may overlap so that the actual category chosen may be a function of 
the situation. So rather than assuming assessment to be a measurement issue, it will be 
treated in this chapter as a categorization issue that involves qualitative judgments. This 
will not be different in the case of other assessment models where behavioral markers 
are used so that raters have to make decisions whether a performance corresponds to 
the description “Establishes atmosphere for open communication” or “Blocks open 
communication” (Flin et al. 2003). The rater has to make a decision between one or the 
other description and, therefore, engages in categorization rather than measurement. A 
decision never is computable, for something computable and therefore pre-determined 
is not a decision by definition. Whereas some readers may think that this leads to a 
less rigorous approach, the following presents several mathematical approaches that 
model assessment based on categorization. As detailed in the case of the fuzzy logic 
approach, the agreement between the predicted and the actual categories is very high. 
This suggests that the outcomes of performance assessment are predictable although 
facts and reasoning involved may be quite fuzzy. 
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The remainder of this chapter describes several rigorous alternatives to the inter- 
rater reliability approach and its assumptions of error variance. Based on the evidence 
presented in Chapters 2 and 3, performance assessment in aviation is taken to be 
categorization—even though there may be some sort of continuity or quasi continuity 
underlying each judgment. Raters assign specific aspects of videotaped scenarios to 
categories of performance rather than doing the equivalent of measuring a physical 
object. Their descriptions of some specific event differ, which leads to different mappings 
onto one or another of the descriptive statements in the instrument. As a result, the scores 
associated with such qualitative assessment as “very poor”, “poor”, “satisfactory”, and 
so on will vary. The descriptions associated with the reasons for a score are actually 
quite fuzzy even though the assignment of a score of 1, 2, or 3 appears to be exact. 
Such qualitative rather than quantitative differences in the observations and reasons are 
exemplified in the case of assessing situation awareness of the pilot flying in the Missed 
Approach scenario (Table 4.1). One may observe, for example, that some assessors note 
that the PF does not predict having to do a missed approach, whereas other assessors find 
evidence in the video that the pilot did talk about the weather and that it might have an 
effect on the flight. Inter-rater reliability approaches can only deal with such variation 
through error variance. Categorization that uses approximate descriptions (e.g., poor, 
satisfactory, and very good), on the other hand, can be meticulously and mathematically 
modeled using fuzzy logic (e.g., Innocent et al. 2004). Fuzzy logic thereby addresses 
the weaknesses of the inter-rater reliability approach (e.g., Ciavolino et al. 2014). In 
addition to the fuzzy logic approach, constraint satisfaction and catastrophe theoretical 
approaches are sketched because these allow understanding such issues as the apparent 
lack of discriminations at a fine-grained level or the role of minor observation differences 
on the ultimate assessment outcome. 


CONSTRAINT SATISFACTION MODEL OF CATEGORIZATION 


Classification may be modeled using constraint satisfaction models (Hutchins 1995a; 
Roth 2001a). In the present instance, different underlying attributes assessors may 
formulate are modeled in terms of nodes that represent the current hypothesis concerning 
the attribute (Figure 4.2a). When information becomes available from the environment 
(an arrow to a node), it feeds into the respective node (taken to represent a hypothesis), 
which has an activation level between 0 and 1. Each observation sentence (see Chapter 
2) that flight examiners make may be considered as an input to a particular node. Some 
pairs of statements support and reinforce each other (+), whereas other pairs of attributes 
counteract one another (—). In the model, one set of attributes (H, to H,) supports a pass 
decision (the activation levels of the nodes | through 6 are {1, 1, 1, 0, 0, 0}), whereas 
another set (H, to H,) supports a fail decision (the activation of the 6 nodes are (0, 0, 
0, 1, 1, 1}). That is, when the first three nodes are active while the remaining three are 
inactive, the network is in a state corresponding to a pass rating; and in the second case, 
its activation state corresponds to a fail rating. Initially, the activations of the nodes 
differ and may start near 0.5 prior to any information. Each piece of information, such 
as those pieces listed in the assessment of situation awareness (Table 4.1), increases 
the activation of the associated node. Once the model is run, the activation levels of 
the individual nodes change under the influence of other nodes; and this therefore the 
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Table 4.1 Sample Situation Awareness Scores for the Pilot Flying in the “Missed Approach and Go- 
around” Scenario and Supporting Reasons. 





Score Reasons 





1 . 


Missed MDA, which is a critical factor; was picked up by the PM on 
missing MDA; 
Missed the correct turn; had to be corrected on the correct turn; 


Did not consider high terrain on the right-hand side where the aircraft was 
turned; 


Plan was configured properly and correctly; plan was at correct speed; 


Did not anticipate go-around, therefore had difficulty predicting future 
events; 


They were at minima and should have considered having to go-around; 
Did not push go-around button; 

Flies against flight director; 

Aircraft flown within tolerances; 

Late in setting missed-approach altitude; 

High workload. 


Definitely comprehended some of the situation; 


It is likely though not observable that the missed approach has been 
debriefed, therefore some uncertainty about predicting future events; 


Set the missed-approach altitude late, it was the PM who picked up the 
missed-approach altitude; 


On the missed approach, it got messy, which puts SA in doubt; 

Did not predict the missed approach, had the mind set on going to land; 
Asked for wind, and did not have correct set up, for the wind arrow 
would have been on the bottom screen, and could gotten wind check on 
finals from ATC; 

Pilot knew being high all along the trajectory; 

Was aware of the cloud in the circling area but did nothing about it; and 
was aware that the conditions are marginal there; 

When realized weather was bad, could have done some active planning; 
should have done something about it but did not project into the future; 
should have been thinking about the missed approach at that time. 


Good perception at all times where they were; 

Talked about the weather, about the effect it might have on the flight, and 
predicted future events; but the PF still carried on; 

While VMC, recognized the huge cloud of turbulence, recognizes that it 
looks dangerous, but still flies into it; 

Recognized that the weather in the circling area ahead is terrible, as 
opposed to pilots who are suddenly and unexpectedly going IMC; 


When confused about the direction of the go-around, announced it right 
away, and then corrected the turn upon input from PM. 





Note: The scores range from 1 (very poor) to 5 (very good). 
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Figure 4.2 (a) In this constraint satisfaction model for a pass/fail decision, different hypothesis 
supporting pass or fail decision draws on information (>) that supports each hypothesis; hypotheses 
supporting and limiting each other are connected by links signed “+” and “—”, respectively. (b) As the 
network settles, its Euclidean distances from the states fully in support of a pass and fully in support 
of a fail change are calculated and plotted. Based on the information available, the end result is a clear 
decision for pass, fail, or a hung state, which may be considered modeling a situation where the examiner 
awaits the outcome of debriefing or where the end result is a “pass with markers”. 


changes the activation level of the constraint satisfaction network as a whole. This 
final state corresponds to one or another observation category: this pilot has failed or 
this pilot has passed (or, equivalently but not developed further here, this performance 
corresponds to a good situation awareness versus this satisfactory situation awareness). 
The movement is similar to that of a network from rubber bands or a steel chain; in both 
instances, the physical situation changes after the system is released until the minimum 
energetic state is reached. (In the case of the chain, the shape will be a catenarian.) 

In the case of constraint satisfaction networks, visualizing the movement is difficult 
even in the case of a simple network consisting of only six nodes. Considering the 
movement is equivalent to simultaneously attending to the changes in the six activation 
levels. That is, it means attending to events in six different dimensions. However, 
the complexity of the problem is reduced when the current state of the network is 
represented as a point in a six dimensional space, and each activation value corresponds 
to the distance from the origin in the corresponding Cartesian grid. The network state 
corresponding to a pass assessment, then, is given when the coordinates of the point 
are given by the vector {1, 1, 1, 0, 0, 0}. This allows us to calculate the distance of any 
state from those corresponding to the pass and fail decisions. Consider the following 
arbitrary starting point where all nodes have the same activation level of 0.5, which 
corresponds to a vector of {0.5, 0.5, 0.5, 0.5, 0.5, 0.5}. The Euclidean distance d between 
this point (network state) and that corresponding to a pass rating is d = V1.5 = 1.222 
In this instance, the distance between the point representing the current network state 
and the point corresponding to a fail rating will also be d = 1.22. 


2 The difference vector d = {0.5, 0.5, 0.5, 0.5, 0.5, 0.5} — {1, 1, 1, 0, 0, 0) = {-0.5, 
—0.5, —0.5, .5, .5, .5}. The length of this vector, using the Pythagorean theorem, is 
ld] =\(-0.5)°+ (0.5)? + (0.5) +(0.5)+(0.5} + (0.5). 
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The initial states of the six hypotheses are set according to the data. For example, in 
the case of the pilot flying in the Missed Approach scenario (Chapter 1), the statements 
“Missed approach is right hand, wasn’t it?” and “It is left?” together with the visual 
information from the video are perceived in quite different ways. Perceiving this as a 
manifestation of an absence of situation awareness is quite different from perceiving it 
as an attempt to confirm the correct direction. We may work on the basis ofa fuzzy scale 
where any value between 0 and 0.15 corresponds to evidence in favor of the hypothesis 
of (near) absent situation awareness. The beginning value of the corresponding node is 
chosen from this range. On the other hand, the second perception may be set between 
0.35 and 0.65, and in fact nearer to 0.65 corresponding to a state of being confused but 
realizing this state and actively mitigating it by seeking further information. 

After each cycle of updating the value of all nodes, the distance of the network 
from the two extreme cases can be calculated. The assessment formation trajectories can 
then be graphed (Figure 4.2b). In this figure, four situations are modeled all of which 
begin with the same initial network state corresponding to the same initial inclination 
slightly toward passing the pilot flying in the Missed Approach scenario. One of the 
trajectories corresponds to the situation where the assessment initially starts with an 
inclination toward a pass rating but ends with a fail rating. Such a situation is reported 
in the case of the Engine Fire on Approach scenario (Chapter 3), where examiners are 
quite confident that the pilots pass the examination until they discover that the scenario 
pilots order evacuation on the side of the running engine. In this case, there is a reversal 
and the flight examiners end failing both pilots. In two instances depicted, the end result 
is a pass decision, in one instance the model arrives at a fail rating, and one assessment 
remains undecided (Figure 4.2b). This latter may be taken as representing the pass with 
marker ratings, which flight examiners used when a performance was not deemed to 
be sufficient for a clear pass rating but insufficient enough to warrant a fail. Or it may 
be taken as a model of a situation where the flight examiner cannot make a decision 
about passing or failing a pilot based on the observations in the simulator and awaits 
the outcome of the debriefing session (see Chapter 7). The differences between the 
trajectories arise either from minor variations in the initial conditions or from differences 
in the connection strengths tying pairs of hypotheses. 

The network model corresponds to the observation that the overall state of 
the documentary sense (current interpretation, inclination) is a function of the 
parts (information about different attributes) but each attribute is a function of the 
documentary sense. This type of model therefore most clearly represents the evolving 
nature of the process of an assessment. In such a model, we can represent all information 
assessors articulate and draw on (e.g., Table 4.1) and treat it in the form of input to 
the corresponding hypotheses. The model then may be used to generate predicted 
assessments and compare these with the actual assessment scores that an assessor or 
pair generates. The model can be improved by adding the information temporally, that 
is, after the simulation has begun, or by making the connection strengths themselves 
subject to change. Doing so would go beyond the present purposes. 

In summary, readers might find it useful to make the explicit link suggested here, 
between the mathematical model underlying the constraint satisfaction network approach 
and the flight examiners’ method described in Chapter 2, which, to some, might appear 
as unprincipled. The model described here thus constitutes an interesting analogy 
that shows how in the context of an emergent overall assessment, some observations 
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(nodes) increase in their activation levels (i.e., become salient and support the decision) 
whereas others tend to zero activation (1.e., observations that are not taken into account 
for the overall assessment). The model therefore constitutes a suitable metaphor for the 
observation about flight examiners methods (Chapter 2). 


CATASTROPHE THEORY MODELS DECISIONS 


Classification, which is based on a true decision between qualitatively different options, 
may also be modeled using catastrophe theory. The mathematician who created this 
theory describes it as a method and language, which, as any language, serves to describe 
reality without being capable of asserting itself as truth or the appropriateness the 
descriptions it generates (Thom 1981). Catastrophe theory is a mathematical approach 
that may be used to describe the existence and emergence of any empirical morphology 
(qualities, types) from quantitative changes. In the natural sciences, a typical case that 
has been modeled using this approach is the transition from liquid water to vapor (gas), 
two qualitatively different states of water. Transitions between binomial situations, 
such as changes in attitudes, conceptual changes of scientists, war and peace, stepwise 
changes in the development of human beings, and category formation among scientists 
all have been described mathematically drawing on this theory. 

One particularly important and frequently used model in the natural and social 
sciences is referred to as cusp catastrophe (Figure 4.3). In this model, one outcome 
variable (vertical) has two values for the same set of values of two independent variables. 
The set of all pairs of values of the independent variables that yield two values for the 
outcome is called the bifurcation set; this set also corresponds to the region below 
the cusp (Figure 4.3). In this region, small, quantitative variations in one or the other 
parameter (independent variable) can lead to a sudden transition from one to the other, 
qualitatively different system state. The approach therefore lends itself to modeling 
situations where some minor change in conditions leads to a major qualitative change, 
such as when a minor issue leads to the breakout of war between two countries that 
have lived heretofore in peaceful co-existence. 

The model only depends on two control variables, a normal factor and a splitting 
factor. When the cusp catastrophe is used to model categorization, the first factor 
corresponds to information available; the splitting factor here are the mitigating factors 
(Figure 4.3). Some researchers use this model in the context of statistical approaches 
(e.g., van der Maas et al. 2003). Eight flags are used to test whether a stepwise change 
describes the data. For example, one flag is the multimodality of behavior. Thus, for all 
value pairs of the two independent variables that are part of the bifurcation set (Figure 
4.3), bimodal distributions in the dependent variable should be observed and normal 
distributions elsewhere. It is apparent that the mathematics associated with catastrophe 
theory is not required at all; instead, researchers use their traditional statistical methods 
to show the existence of bimodal distribution for the phenomenon of interest. Similarly, 
the hysteresis flag marks the fact of sudden, necessary jumps at the edges of bifurcation 
set: the two observable jumps are in opposite direction, from the lower to the upper 
plane and vice versa (Figure 4.3). It is apparent that this statistical approach requires 
a significant number of cases making transitions between two states under the same 
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Figure 4.3 Cusp model of pilot assessment, where an initial appreciation considers the performance 
a pass but where, with detailed assessment, there is a sudden transition to a fail rating. Within the 
bifurcation set, both pass and fail decisions are possible states and transitions from one to the other 
may be provoked by minute variations of information. 


or similar conditions; and it is apparent that the phenomenon is only thought of as 
exhibiting the catastrophe theoretic characteristics but the model is a statistical one. 

The cusp catastrophe is useful to describe many phenomena in the natural sciences. 
In the way presented, that is, with continuous independent variables, the model is less 
usable in psychology and other social sciences (Hu and Xia 2015). This is so because 
behavioral and independent variables tend to be discontinuous and categorical and are 
expressed in language that is inherently and irremediably indeterminate, incomplete, 
and fuzzy. This is quite apparent when the observation made by some flight examiners 
(e.g., “the captain in the Limited Fuel Reserves scenario leans on the first officer”) is 
contrasted with the observation sentences of other flight examiners (e.g., “the captain 
tries to convince the first officer” or “the captain encourages the open discussion of 
the alternatives). These examples show that the information variable in this case is 
discrete rather than continuous. That is, expressing variables along continuous scales 
does not correspond to the observations; not only is the expression along continuous 
scales difficult but also is it arbitrary. 

The assessment of pilot performance in essence rests on categorization, which 
in the simplest case qualifies a pilot as having passed or failed an examination. Flight 


Modeling Performance Assessment 115 


examiners, in the course of observing, may even flip-flop between leaning towards 
passing and failing pilots. A number of situations have been observed in the field 
where flight examiners did not know whether they would pass or fail pilots until after 
having had the opportunity to debrief them. Whatever qualitative information they 
obtained in the debriefing session then swayed the decision in one rather than the other 
way. Flight examiners might subsequently explain that they arrived at a decision as 
soon as they had found out what the pilot in question had been thinking. That is, the 
movement is from finding out something, that is, from receiving a bit of information, 
to the decision between two alternatives (pass, fail). Clearly, this is not a process along 
some continuous scale. 

In a discrete approach to catastrophe theory, such outcome variables are modeled 
by a function F defined in terms of pairs of qualitative and discrete values: F = (q, d), 
where q = {pass, fail} (or, numerically, {+1, —1}) and d = {inc, std, dec} represents 
the direction of the change between two points in time—increase, steady, decrease, 
respectively. In the same way, the values of independent variables are defined in terms of 
fuzzy categories: U = (m, d) and V= (n, d), where m, n = {very low, low, normal, high, 
very high}. In this way, it is possible to model the different observations flight examiners 
make about behavior of the Limited Fuel scenario captain in qualitative categories 
between the extremes “very open discussion” to “leaning on” or “trying to force” the 
first officer. Using the idea that categories have a more definitive core surrounded 
by fuzzy edges, each observation—e.g., (g, d) = (fail, inc)—can be transformed 
into a four-tuple representing a fuzzy value. Thus, for example, the value (fail, inc) 
=[-.2, 0, .2, 0] indicates a definite categorization core from —.2 to 0, with a hard boundary 
to the right (i.e., at 0) and a fuzzy boundary from —.2 to —.4. The same is done for the 
two independent variables. The fuzzy, qualitative distinctions—e.g., between very low, 
low, and normal—are extracted from the verbal protocols where assessors give reasons 
(see section on fuzzy logic below). These data then can be used in conjunction with a 
software package such as QSIM. 

The category formation case is especially useful in modeling the assessments flight 
examiners make at the boundary between a pass and fail rating; it could equally be used 
to model the distinction between two adjacent ratings of a human factor category, such as 
“poor” or “minimum standard” versus “satisfactory” performance from the perspective 
of situation awareness. This model is consistent with the observation that the assessment 
at the boundary between two alternatives (e.g., pass/fail or poor/satisfactory) remains 
difficult because of the fuzziness and indetermination of linguistic expressions that 
record observations; and minor variations in expressions may indeed precipitate a sudden 
transition in the outcome variable. This sudden transition following minor variation 
models, for example, situations in which flight examiners actively look for those tiny 
pieces of evidence that allow them to pass or fail that reflects their overall sense. The 
approach also takes into account that flight examiners may be resistant to moving to 
particular scores, for example, resistant to attributing a score of 1 or resist providing a 
score of 5. The shortcoming of this approach is that it reduces the input to two variables 
rather than taking into account all the observation sentences that are articulated during 
assessment and therefore have some bearing on the assessment outcome. This aspect 
is actively addressed in the fuzzy logic approach to categorization. 
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FUZZY LOGIC 


In the preceding section, the idea has been introduced to use fuzzy logic for modeling 
(a) the irremediably indeterminate, incomplete, and ambiguous nature of language and 
(b) the discontinuous, categorical, fuzzy-edged nature of everyday classification. The 
case of assessing situation awareness in the Missed Approach scenario (Chapter 1) is 
used to exemplify this approach to modeling the assessment of pilot performance. As 
shown in Chapter 3, there is considerable variation between the different assessor pairs 
across the six performance categories, which often range over 3 and up to 5 points on 
the five-point scale (Figure 3.3). The inter-rater agreement across 36 rating instances 
is fair in only one instance (Figure 4.1) and poor in all other instances. With respect 
to situation awareness of the pilot flying in the Missed Approach scenario, the ratings 
range from a low of | (very poor) to 3 (satisfactory). The variations are especially 
interesting given the fact that these scores already reflect consensus within the rater 
pairs (a practice some airlines use to reduce variability), which should have a regressive 
effect given that fewer important facts are missed and more extreme evaluations are less 
likely. To understand these variations in the ratings, the transcripts of the assessment 
sessions were analyzed to reveal the reasons that each pair provided for grounding its 
assessment decision. 


Description of the Data 


The assessment scores are 1 < SA < 3, with a mean of SA = 2.44 (SD = 0.73) and a 
mode of 3 (Figure 3.3). A previously reporting modeling effort extracted from the 
transcriptions all comments with respect to specific contexts of the flight generally 
linked to the discussions of situation awareness (Roth and Mavin 2013). To support 
statements about the pilot flying’s (PF) situation awareness assessment, the reasons 
include observation descriptions from three aspects of the scenario as shown in the video: 
(a) the current weather situation in the area where the aircraft was to make its final turn 
to land during the circling approach as seen from a distance of about 5 km (see Figure 
1.1); (b) the proposed or actual turn direction taken by the PF immediately following 
the identification of the IMC condition; and (c) the awareness of the flight parameters 
on the glide path during the approach prior to reaching minimum descent altitude (see 
the box for the exchanges between PF and PM in the first two aspects). Whereas the 
model normally includes all observation statements separately, for ease of presentation 
and intelligibility, the remainder of this discussion here only uses these three areas. 





Missed Approach Excerpt 


Regarding the weather, the crew is talking about the wind speed and direction at the 
level of the aircraft and on the ground. At that time, the aircraft is at the minimum 
descent altitude and about 1 minute after having become visual with the runway 
(Figure 1.1). The pilot flying (captain), looking out the window to his left, and 
comments on the weather: 
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Scenario excerpt 4.1 


01 PF: The rest doesn’t look too crash hot out to the east so we’ll have to be 
careful? 
02 PM: Yes. Just dropping the range scale down on the terrain display. 


About 75 seconds after the exchange in this scenario, just as the PF begins to 
circle, the crew loses visual reference to the runway when they fly through a rain 
shower. The following exchange occurs. 


Scenario excerpt 4.2 





01 PM: What? 

02 PF : Going IMC. 

03 PM: Rain shower there. 

04 PF : Missed approach was right hand, wasn’t it? 
05 PM: Negative. LEFT hand, LEFT hand! 

06 PF : Itis left? 

07 PM: LEFT hand. 











The actions and reactions of the PF to the weather situations constitute the ground 
for the observation statements that performance assessors used in the reasons for making 
an assessment of situation awareness. The perception of the weather situation in the 
circling area is an important aspect of the assessment discussions that mitigates the 
assessment of situation awareness and decision-making. The comments concerning 
the weather show, however, that the same perceived facts lead to different assessments 
as to the level of situation awareness (Table 4.1). In all but one rating situation, the 
fact is noticed that the PF perceives the poor downwind weather situation in the area 
where the circling will occur. Observations include that the PF may have difficulties 
comprehending the weather situation, translating the observation into actions “inside 
the cockpit”. Especially, the captain is seen to fail predicting future events, that is, he is 
seen as failing to make a link between the observation of bad weather and the possibility 
of a missed approach. From the reasons in the assessment, we may construct a model 
for how assessments are composed; this model shares some essential features with a 
theoretical model of situation awareness (Endsley 1994b), not in the least brought about 
by the three same components of the category (Figure 4.4). 

In the model of the assessment process, observations made and articulated are 
used as the basis for considering the different aspects of situation awareness. In some 
instances inferences are used as the basis for commenting on a component. Thus, for 
example, based on the observation that the PM does not articulate the possibility of 
having to go around, an inference may be made about comprehension of the weather 
perceived. In other assessments, an observation might include that the PF states having 
to be careful, which is taken to manifest projection. But then it may be noted that the 
PF “does nothing about the weather”, which means that whatever the output from 
situation awareness, it does not affect decision-making as shown in the model (Figure 
4.4). Given that situation awareness involves perception of facts, their comprehension, 
and prediction of the future, it is perhaps not surprising that its assessment ranged from 
unsatisfactory to satisfactory (Table 4.1). 
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+ sees the weather + says “we have to be carefu 
+ asks for wind - is not predicting go-around 
+ has a gut feeling of what is going to happen 
+ is seeing things for the future 


- didn’t think having to do go-around - does nothing about weather 
+ comprehended situation - just carries on 
- is surprised that they go IMC 
- has to ask for correct turn 
Observations 


Figure 4.4 In assessing the weather situation in the Missed Approach scenario, observations (objective 
sense) made by viewing the videotaped scenario ground an operating model of pilot cognition, which 
is derived through the documentary sense. Observations may weigh positively (+) or negatively (—) 
on the documentary sense for the three components that make up situation awareness: perception, 
comprehension, and projection. 


With respect to the events later in the scenario when the aircraft begins the circling 
maneuver, assessors find evidence in the video that the PF initially initiates a slight right 
turn, “rolling the wrong way”; there are suggestions that the captain “was initially going 
to turn the wrong way”. Sometimes what is happening is documentary evidence for pilot 
intention without actually stating that the aircraft had started rolling to the right: “going 
to turn the wrong way [right]” and “want[ing] to go right”. Finally, assessors may note 
that there is “a bit of confusion” and that “rather than just do the wrong things”, the 
captain asks or confirms whether the turn is right before actually turning. Observations 
may include fuzzy statements (e.g., “there is confusion as to whether it is a left or right 
hand”). In general, whenever it is observed that (a) the aircraft is actually rolling to 
the right or (b) the captain wants to go right before the first officer corrects then this 
observation weighs heavily in the assessment of situation awareness (downward). 

In the assessment of situation awareness, it is generally noted that the captain 
appears to be unaware of the fact that the aircraft was between 60 feet and 200 feet 
higher than the glide path specified on the approach chart and generally 20 knots above 
his reference speed; some observers may characterize the flight parameters to involve 
“big fluctuations”. Assessors also may note that the PF fails to appreciate the wind from 
behind and the effect that this has on the glide path of the aircraft. It may also be noted 
that the PF is unaware of the minimum descent altitude, which a pilot should always be 
aware of to guarantee a safe landing. To a lesser extent, other reasons also are used to 
ground assessments. It may be noted that the circling approach was dangerously close 
to the mountain range and that there has been a crash with loss of life in the past. The 
captain therefore should have been aware of the danger and should have been prepared 
to fly the correct missed approach procedure given the weather conditions. 
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Chapter 3 shows that assessors do not observe precisely the same facts, for example, 
in the case of the turn (to be confirmed, intended, actually enacted). Given the different 
facts noted and the different degrees to which observed facts entered the assessments, 
it may not surprise that assessments come to different conclusions about how to rate 
the level of the captain’s situation awareness. Readers should note that all facts and 
reasons are stated in irremediably imprecise language. The following shows how the 
final ratings can be modeled and thus explained despite the imprecise observational 
and conceptual language. 


Modeling the Assessment of Situation Awareness 


Raters observe different facts and weigh the observed facts differently. In this chapter, 
given the problems with the inter-rater reliability approach, assessment is treated and 
modeled as a categorization issue where multiple, inherently vaguely defined and 
ambiguous categories and terms are in play. That is, the purpose here is to provide a 
way of arriving at a categorization of performance given the facts and reasons as well 
as their weights that enter the assessment. This is similar to modeling the diagnostic 
efforts of doctors based on the symptoms that they identify and describe in fuzzy 
terms. The approach is premised on the understanding that flight examiners and others 
in pilot performance assessment have good albeit imprecise reasons for grounding 
their assessment-related decisions; and the approach is consistent with the research on 
categorization, which shows that most categories are not hard and fast but are separated 
from neighboring categories by fuzzy regions (e.g., Lakoff 1987). The fundamental idea 
is to model how the everyday, uncertain, imprecise, and underdetermined descriptions 
assessors (initially) make lead to specific categories of performance that they might 
express by means of performance categories very poor, poor, satisfactory, good, and 
very good. In this way, no assumption has to be made about any quantitative continuity 
implied in a measuring instrument. Assessment then can be modeled by means of fuzzy 
cluster analysis. 

In their assessment, raters categorize performance into one of five possible 
categories. Each performance category is described by two matrices, one describing 
the lower and the other the upper bounds of the five categories. These bounds represent 
a shared value system such as it might be specified in a performance-describing 
document. The whole spectrum from very poor to very good is distributed over [0, 1]. 
Thus, for example, one study proposes the lower and upper bounds for very poor to 
be 0 and .2, for poor 0.2 and 0.4, and so on (Roth and Mavin 2013). Two other studies 
define the boundaries for very poor (vp), poor (p), satisfactory (s), good (g), and very 
good (vg) as {0 < vp < .15 < p < .35 < s < .65 < g .85 < vg < 1} because performance 
raters have been reported to tend to avoid extremes and exhibit a preference for the 
mid-range score (Roth and Mavin 2015; Roth et al. 2014b). As can be seen, all studies 
use hard boundaries where two categories touch. However, categories do not have hard 
boundaries; instead, as shown in the comparison between cup and mug, there are grey 
areas. Any object in a grey area may be assigned to either category—e.g., as this is the 
case in the catastrophe model above. Clear distinctions are hard to make if at all. In the 
preceding section, we see that the boundaries may overlap (hysteresis) and, in addition 
to an actual rating, there may be a tendency toward a lower, higher, or same category. 
For example, situation awareness may be assessed as “poor”, and there is evidence of 
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resistance to move to the very poor category despite some of the observations might 
justify. On the other hand, there might be resistance to moving toward the higher quality 
because of some crucial observation. In the Engine Fire Before Landing scenario, the 
observation that the aircraft was evacuated in the direction of the running engine was 
taken to be very serious, so that there may be resistance to rate situation awareness 
as “poor”, which would have been legitimized by the fact that all other aspects of the 
approach manifest good and even very good awareness for the situation. This fuzziness 
and possible overlap of the five performance categories therefore is modeled here by 
means of an overlap of the boundaries. This yields the following matrices describing 
the lower (B,) and upper bounds (B,) of the five categories for the three types of 
observations: weather (W), turn (T), and glide path (GP): 


W T GP W T GP 
0 0 0 | very poor „175.175 .175| very poor 
.125 ,125 .125 poor 375 .375 .375 poor (4.1) 
B,=|.325 .325 .325 |satisfactory, B,=|.675 .675 .675 |satisfactory 
.625 .625 .625 good .875 .875 .875 good 
.825 825 .825 | very good 1 1 1 | very good 


In this way, an association is made between a continuum and a set of categories 
(e.g., Bharadwaj 2007). 

We also define a fuzzy relation W, which specifies the pertinence of a specific 
behavior in the assessment of the matrix of each component. This matrix, too, represents 
a norm, which is specified based on the general observations of the importance of 
weather, turn, and glide path to the scenario. The weight matrix might be given by 


vp p s g vg 
9 9 8 9 IW 

W=|1 9 9 9 1|T (4.2) 
0 2 .5 8 1|GP 


where each number signifies the weight of a given observation {W, T, GP} fora particular 
level of performance {vp, p, s, g, vg}. Thus, for example, for a rating very poor (vp), 
how the rated pilots deal with the weather has a weight of .9, how they deal with the 
turn has a weight of 1, and their awareness of the glide path parameters do no figure at 
all (weight = 0). For receiving an absolute perfect score on situation awareness (vg), 
all aspects have to be perfect (i.e., W T, GP = 1). 

We now need to identify the sets of observation that are used in support of an 
assessment decision. Even such ephemeral descriptions as the relation between two 
pilots are modeled by means of fuzzy scores. Thus, for example, in the context of 
communication in the Limited Fuel Resources scenario, assessments often invoke 
(a) degrees to which the captain is leaning on the first officer to get a particular viewpoint 
accepted, (b) a certain degree of pressure, (c) a first officer who is sticking to his guns, 
(d) a first officer who does not succumb to the captain’s pressure, and (e) the very 
subtle ways of using body language to pressure a first officer all the while saying, 
“T don’t want to pressure you”. Degrees of “leaning on” now can be defined as a fuzzy 
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set and used to model a non-technical skill (human factor) such as communication. 
A first decision is made as to the membership function, that is, the boundaries for a 
particular quality of the relation between the two pilots. For example, the data may 
suggest three categories in the case of the captain: strong pressure, subtle pressure, and 
open dialogue are to be distributed over [0...1]. The data may justify the decision to 
use .33 and .66 as the points dividing two adjacent membership categories, all verbal 
statements however imprecise may then be assigned a score at the center of a category 
with adjustments made for the tendency. In this way, “subtle pressure” corresponds to 
a score between .33 and .66, investigators might decide to begin with .5 as a tentative 
score, and then adjust downward or upward depending on the reasons stated. Thus, the 
observation about body language and the statement “I don’t want to pressure you” tends 
toward more open dialogue (thus warranting a score of .55 or .6) than the observation 
“a certain degree of pressure” (warranting a score of .4 or .45). 

Let us now assume that a particular set of observations is used to justify the 
assessment of situation awareness, each consisting of a specific behavior (“symptom”) 
W (appreciation of downwind weather situation), 7 (turn following the missed approach 
call), and GP (glide path parameters). This set is translated into a vector expressing a 
particular performance level A = [W, T, GP]. In the choice of the particular value of 
an observation such as W, all statements pertinent to weather are considered. To keep 
the description transparent, consider the following case. The pilot flying is observed 
as perceiving the weather but there is no evidence that the weather situation is actually 
understood; but there also is no evidence that it is not understood. However, assessment 
reasons may note that the pilot probably has some difficulty predicting what might 
happen given the cloud in the circling area. The situation is described in terms of a pilot 
having to be careful of the weather and see it coming. It is noted that the pilot flying 
identifies the cloud and associated risk but still ends up entering the cloud and being 
surprised going IMC. The assessment might conclude that “although there are a few 
errors”, the pilot is “still quite good overall” and only the “comprehension a little bit 
down”; for this reason, the assessment “must not be too harsh”. The weather variable 
therefore is set to W = .5. The assessors note that there is a brief turn to the right, which 
is mitigated by the fact that the crew corrects the error. Although the action is not really 
satisfactory, mitigating factors are cited with a tendency toward the positive: the turn 
parameter is therefore set to T= .4 (in the lower half of satisfactory), which reflects the 
fact that the turn is wrong but is corrected. Finally, the assessors note that the glide path 
parameters are reasonable throughout the scenario, leading us to set the parameter to 
GP = .75 (in the upper half of the good category). The fuzzy set of observations used 
in this situation therefore is 


W T GP 
A=[5 4 .75] 


To model the assessment, a clustering technique is used. It determines that 
assessment category with which the assessment based on the observations is most 
similar. The distance between the raters’ fuzzy set of observations and the performance 
categories on each of the observed context (1.e., the diagnostic cluster specified in 
eq. 4.3) (Klir and Yuan 1995) is calculated according to 


(4.3) 


D= p [WB -AJF +E [W B,- AF] i (4.4) 
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which fundamentally is a Euclidean metric commonly employed in clustering techniques. 
Concretely, then, in the case of the assessment with the preceding observations we obtain 





D = ([0.9(0-.5)P + [1(0-0.4)]? + [0(0-0.75)]? + [0.9(0.175-0.5)]? + [1(0.175—0.4)P 
+ [0(0.175-0.75)1?y2 





as distance between actual observation and performance specifications for an 
unsatisfactory grade. The calculation yields D = 0.80. The same calculations are done 
for all other categories, poor to very good yielding a total of five values (see Rating 5 
in Table 4.2). 

Another assessor (pair) is quite adamant about the glide path parameters and the 
turn: thus, two things a pilot should be aware of include (a) minimum descent altitude 
that the aircraft is descending to and (b) the way in which to turn after the missed 
approach call (especially important given the high terrain to the right of the aircraft). 
These two observations underlie the documentary sense that there are “two gaping 
holes in situation awareness”. Concerning the turn, we consider the pilot flying as 
“going to turn right” rather than as actually turn right. Glide path parameter and turn 
are therefore set at T= 2 and GP = .1 in this assessment scenario. Weather is another 
issue that the pilot flying is not aware off in this example, especially in anticipating 
the possibility of a missed approach. The weather parameter therefore is set to 
W = .1. Conducting the same calculations as before for the unsatisfactory, minimum 
standard, and satisfactory categories yields the first rating scenario (Table 4.2). In all 
but one scenario, the assessments are based on all three areas, that is, the assessment is 
modeled by a vector A=[W, T, GP]. In one instance, the assessment does not include 
any statements about the weather. The fuzzy assessment vector A = [7, GP] models 
translates the fuzzy verbal statements. 

In this approach, the minimum distance indicates the category predicted by the 
fuzzy logic model; a box surrounds the minimum distance for each case (Table 4.2). 
In this particular instance, a comparison between the predicted rating and the actual 


Table 4.2 Actual and Predicted Ratings of Situation-Awareness for the Pilot Flying in the Missed 
Approach and Go-around Scenario. 

































































































Rating Actual Predicted Rating 
Scenario | Rating 
1 3 5 
(Satisfactory) (Good) (Very Good) 
1 1 0.73 1.34 2.25 
2 2 0.54 1.08 1.83 
3 2 0.52 0.47 0.97 2.25 
4 2 0.44 0.47 0.87 2.25 
5 3 0.80 0.52 2.25 
6 3 0.97 0.45 2.25 
7 3 0.90 0.60 2.25 
8 3 0.80 0.66 2.25 
9 3 0.94 0.70 2.25 
Note: A box marks the predicted minimum distance; grey shading indicates agreement between actual 


and predicted score. 
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rating gives a perfect fit. Modeling decision-making in the same Missed Approach 
scenario (without overlapping categories) yields on incorrect categorization. In this case, 
assessments included between two and five areas of evaluation. This result is reproduced 
if the current approach is applied. A study investigating the rating of knowledge and 
procedures in the Limited Fuel Resources scenario (see Chapter 3) also reported a perfect 
fit (Roth et al. 2014b). In that instance, the model translates statements about how the 
captain in the scenario deals with the standard operating procedures with respect to the 
limited fuel resources. The assessors’ observation statements ranged from “very hard”, 
such as that the captain compromises the resources to softer ones, which that the captain 
is (a) proposing something legal though risky, (b) creative in the interpretation of the 
standard operation procedures (SOP), (c) reading between the lines and blurring the 
boundaries, (d) taking a poetic license to the SOPs, and (e) aware that there are ways 
to fudge the SOPs. 

To evaluate the sensitivity of the model with respect to variations in the fuzzy 
scores assigned, the model was tested by varying one of the two parameters that heavily 
weighs on the assessment of situation awareness: weather. The predicted categories 
were recalculated, once with the associated fuzzy score decreased by 0.1 and once with 
the fuzzy score increased by 0.1. In the present scenario, the categorization remained 
perfect with the lower scores. However, in two cases, situation awareness was classified 
incorrectly by moving to a more favorable assessment: from very poor to poor in one 
instance and from satisfactory to good in another instance. These results show that the 
fuzzy approach is quite robust with respect to variations in the fuzzy scores assigned 
to the observation sentences that assessors make. 

A case such as the weather and its implications for the flight safety, here set by 
holistically choosing a single parameter value, is modeled as a composite in a more 
realistic approach. Such a more realistic approach takes into account the following. 
Fuzzy codes are assigned to statements about the pilot’s perception of the weather in the 
area of the final turn, its comprehension, and associated projections for the flight. There 
may be more types of observations, each of which is coded. Each of these dimensions 
is associated with fuzzy descriptions such as low, acceptable, and high; these values 
then would be translated using a membership function derived from the data. Similarly, 
the awareness of glide path parameters is modeled as a composite, each of which is 
associated with fuzzy evaluations (e.g., unacceptable, acceptable, and accurate). The 
turn parameter is the result of assessment observations that there is (a) a confirmation 
of a left turn (medium, medium-high awareness), (b) an intention to turn (poor), and (c) 
an actual right turn (very poor). All of these changes improve the predictive accuracy 
of the model. 


TOWARD A NEW CONCEPTION OF PILOT PERFORMANCE 
ASSESSMENT 


The International Civil Aviation Organization makes considerable efforts toward global 
quality assurance. Integral to these efforts is the reliable and valid assessment of pilot 
performance, including the crew resource management (CRM) skills. As shown above, 
some authorities make inter-rater reliability a core requirement for the inclusion of 
specific CRM skills in assessment. Reliable assessment as established in the traditional 
approach to variation in score assignments, however, continues to be a difficult area in 
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the aviation industry. In this first part of the book, evidence is described and theorized 
that explains why efforts toward greater inter-rater reliability may be doomed. Flight 
examiners and other pilots tend to have very good, intelligible, plausible, and convincing 
observation-based reasons for each assessment they make. There are good reasons for 
scores to differ, good reasons on the part of the assessors and good theoretical reasons. 
As to the assessors, they make observations that enter the assessment, which the fuzzy 
logic approach presented here models quite well. There are good theoretical reasons as 
well, for observations show that assessors (a) note qualitative rather than quantitative 
differences, (b) use categories that irremediably have fuzzy boundaries, and (c) express 
observations in inherently ambiguous language. 

Much of the effort conducted so far to increase inter-rater reliability has been 
devoted to developing methods that decrease variation attributed to rater error— 
including rater training. The underlying assumption is that assessment is a measurement 
issue and decreasing the error through training can reduce related error variance. Few 
studies have investigated different ways of thinking about assessment: raters tend to 
have good albeit imprecise, language-mediated reasons (heuristics) for making their 
assessments. Moreover, the categories they are required to assess, as most human 
categories, are not hard and fast but fuzzy; and the specific observation made and the 
constitutive relation these have with theoretical categories introduce further fuzziness. 
By choosing such a perspective we acknowledge and build on the inherent rationality 
of human endeavors in the face of the fuzzy nature of language and categories. The 
approach presented here models these endeavors even in the face of uncertainty, 
ambiguity, and variations with which assessment objects can be perceived, described, 
interpreted, or appreciated. In this chapter, three approaches are presented that model 
how observations and imprecise descriptions lead to specific assessment categories 
that artificially are mapped onto continuous rating scales (e.g., 1, very poor, to 5, very 
good). In the case of fuzzy logic, the approach articulated here in greater detail, very 
good agreement is achieved between the predicted and actual assessment scores. This 
also intimates that the conceptual model of assessment is quite good. 

In part the very good agreement between model and actually observed assessment 
scores arises from the fact that the concrete though fuzzy descriptions humans use in 
describing the cases they are asked to classify. The model presented here therefore 
does not presuppose that all assessors rate the same objective case but instead that they 
rate the object in the way it appears to them within their perceptual horizons. This, 
therefore, corresponds to the transactional approach where agents (here assessors) and 
their environment (here the flying scenarios assessed) constitute a single unit. We may 
articulate the result in this way: Zf we account for the fact what the assessors actually 
rate, then the predictability of the ratings is very high. On the other hand, if we assume 
that all assessors rate the identical object—an assumption that is clearly false, as seen 
in Chapter 3—then the predictability of the ratings decreases considerably. 

In this chapter, an example is provided of how to model the assessment of situation 
awareness. The fuzzy logic approach is highly promising, as it affords understanding 
why and on what grounds different assessors arrive at quite different assessments of 
the same video clip. Although some authors suggest that there is no or very little value 
to concepts such as situation awareness (e.g., Dekker 2015b; Dekker and Hollnagel 
2004) there is not only usefulness in non-technical performance descriptors (such as 
situation awareness) but also a rigorous approach for predicting how raters will assess a 
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situation given how they understand specific parts of the assessment object. In this case, 
rather than abandoning the existing human factors-related language used in the industry, 
we might be better off thinking about how to use the observed variations positively 
to (continuously) improve pilot performances. In Chapter 10, a case is described that 
makes apparent that situation awareness can be an index that gives a particular, useful 
focus to the instructional efforts that it gives rise to. 

If assessors of pilot performance have good albeit imprecise reasons for varying 
in their assessments—1.e., justifying the variations—then there may be little hope in 
improving inter-rater reliability. We now understand and can model these variations. 
This should encourage researchers and the industry to rethink recurrent training and 
assessment. Less effort might be spent on assessment as such and more effort should 
correspondingly be devoted to performance improvement and training. For example, 
we might begin to think using the diverse and divergent descriptions of performance 
as starting points for conversations that might lead participating pilots and examiners 
to say things like “What an interesting way to think about what happened there?” or 
“T have never thought about doing this in such a situation!” The industry may then 
move from an orientation toward assessment to a focus on learning. In some airlines, 
the culture appears to be almost entirely focused on assessment with little effort to 
discuss in details actual simulator events for the purpose of learning and improvement 
(Chapter 5). On the other hand, especially when a debriefing tool is available, attention 
toward pilot learning and professional development may be increased. Thus, an 
examiner’s observations, descriptions, and explications in terms of human factor 
concepts (e.g., situation awareness, communication, or management) might serve as the 
starting point for discussing performance for the purpose of triggering learning events. 
How such debriefing meetings unfold and the functions that a variety of representational 
tools may have are the contents of Part B. 
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PART B 


COGNITION, TECHNOLOGY 
AND WORK IN DEBRIEFING 
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n the everyday world of sensuous praxis, order is often implicit, produced and 

articulated for the purpose of getting done the job at hand. To use an everyday 

example, a queue (line up) is a social fact that we not only observe but also 
contribute to making (e.g., by lining up at the checkout counter of the supermarket). 
We do so without having explicit rules for queuing and we never have to learn rules 
before participating in making queues and making visible when there is a problem. By 
asking “Is this the lineup for movie X or Y?”, we indeed formulate that the tails of two 
queues overlap and that it is unclear where to line up for one or the other movie. We 
contribute to producing queues as if we were acting according to a plan even though we 
do not have one. This order that we produce and formulate for one another in mundane 
everyday praxis is referred to as mimesis. Both piloting an aircraft (Chapter 1) and 
debriefing (Chapter 5) constitute cultural activities in which people are caught up 
doing their everyday work without having to talk about and theorize it in any reflective 
manner. Part A describes and theorizes the work and cognition of flight examiners who, 
based on their observations of simulator sessions and while debriefing pilots, produce 
pilot performance assessments. That is, the first part of this volume is concerned with 
exhibiting how the order observable in pilot performance assessment is produced, that 
is, order as it appears in observations as much as explanations that make use of human 
factors. On the other hand, Part B is concerned with the orderly production of order: 
debriefing is an organized and orderly societal activity in which order is brought to the 
generally inchoate lived-experiencing of pilots while undergoing recurrent training and 
assessment in their simulator-based tasks. The following two sections briefly take a look 
back at existing work and meta-analyses of studies on debriefing and then introduce 
and provide an overview of the second part of this book. 


LOOKING BACK 


Debriefing is a societally motivated and socially organized cultural practice for reflecting 
on and reviewing happenings that have come to a close and thereby have become 
definitive, namable events. That is, debriefing always concerns something that has 
occurred and that already has been completed. The purposes of debriefing the agents 
that were involved in the events include improving the future performances of these 
agents when they find themselves in the same or similar conditions. Debriefing is used 
in many areas of society but is particularly practiced in fields with high risk to agents 
or their clients, including the military, the medical field, psychological practice, nuclear 
industry, and medical education. Despite the importance of debriefing to professional 
learning and assessment in a broad range of professions, a recent study notes that “there 
are surprisingly few papers in the peer-reviewed literature to illustrate how to debrief, 
how to teach or learn to debrief, what methods of debriefing exists and how effective 
they are at achieving learning objectives and goals” (Fanning and Gaba 2007); another 
review article suggests that the theoretical and empirical human factors literature 
concerning debriefing is sparse and fragmented (Tannenbaum and Cerasoli 2013); and 
the problems of debriefing tend to be attributed to implementation (e.g., Zuckerman 
et al. 2012) without regard to the contextual particulars. This lack of research also is 
reflected in the field of aviation, where one may find many assertions about learning that 
accrues from “experience” in the absence of any theory about cognition in debriefing, 
how pilots remember, what the function of narrating past events is, and how the work 
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of debriefing captures the preceding work of flying an aircraft (simulator). There is 
also a complete lack of an theorization of the concept or category of experience and 
of the differences between the lived-experiencing characteristic of agents who do not 
know what will have happened when everything has been done and the definite nature 
in which agents state that they have had an experience, which inherently requires some 
end to have occurred so that the experience as a whole can be grasped. 

The practice of debriefing, which has its likely origin in the military, constitutes a 
means to learn from what has happened in the past for the purpose of developing new 
strategies by reviewing, analyzing, and discussing pertinent (real or simulated) events. 
Debriefing is useful especially in aviation where, as in other fields with high-stakes 
environments, errors can have considerable, often deadly consequences. Although 
searching for literature using the term debriefing may turn up many references, the 
above-cited reviews and meta-analyses of the scholarly literature suggest shortcomings 
in the topics researched, paucity of related theory, limitations in the number of empirical 
studies, and problems in research design. There appears to be general agreement among 
existing reviews and meta-analyses that the practice of debriefing tends to be beneficial, 
as indicated by some studies with positive learning and performance improvement. 
Simultaneously, there also is agreement of considerable variation of the findings of 
studies included in the meta-analyses. 

Existing studies investigate debriefing by focusing on the words of individual 
pilots describing what they have done. There are suggestions, however, that what 
happens during a flight episode needs to be considered in terms of the cockpit as a 
joint cognitive system, involving humans, automatic pilot, and material setting (see 
Chapter 1). In part consistent with this suggestion are the meta-analytic findings that 
report greater effectiveness when team rather than individual performance is reviewed. 
However, there are also suggestions that team debriefing is preferable when the objective 
is improvement of team effectiveness, whereas debriefing of individuals is preferable 
when the goal is enhancement of individual effectiveness. 

Although there frequently is a focus on the technology available in debriefing, 
especially videotapes, the cited meta-analyses consistently report that the outcomes of 
video-mediated debriefings are not different from debriefings without video. However, 
analyses how the viewing of videotapes changes debriefing practices and the effects 
these might have on cognition could not be found. This may be so because intervention 
studies that focus on pre- and post-intervention measures tell us little about how 
debriefing unfolds, what the cognitive demands are, how participant engage in and 
constitute the cultural practices of debriefing, how past events come to be represented, 
and so on. It is not surprising that the cited meta-analytic studies are hard-pressed as 
to the reasons for the lack of outcome differences between debriefing meetings using 
versus not-using videos. 

One existing study of debriefing in aviation, conducted with five U.S. airlines, 
provides information in four areas: instructor pilot facilitation and crew participation, 
effects of crew position and crew size, content of discussion, and duration of meetings 
(Dismukes et al. 2000). First, the study suggests that the instructor pilots with 
two-member crews tended to talk more than the crewmembers taken together, often 
asking questions, with little within-crew talk. Second, the first officers and captains 
contributed about the same amount of talk. Third, there was considerable variation 
between airlines as to the number of words concerning crew resource management 
and technical topics. The remaining talk was classified as mixed and non-specific. 
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On average, 41% of the instructor pilot talk and 52% of the crew talk concerned crew 
management and technical performance of the crew. Finally, the debriefings lasted 
on average 31 minutes, with a range of an order of magnitude between the shortest 
and the longest meetings, which is almost twice the 17.85-min duration that a recent 
meta-analysis reported for all the 111 studies included (Tannenbaum and Cerasoli 
2013). In the case of aviation, really thorough discussions tend to occur in one-hour 
meetings (Dismukes et al. 2000). These authors also note that only one instructor in 
their research attempted to have crew-led discussion. The study concludes that greater 
crew involvement and control would be desirable following line-oriented flight training 
(LOFT) sessions. 

A major shortcoming of existing work on debriefing in education may be the 
virtual absence of theory attempting to capture what is happening when pilots and a 
flight examiner or other facilitator are debriefing events that have occurred in the flight 
simulator or on line flights. Thus, the edited volume Facilitation and Debriefing in 
Aviation Training and Operations (Dismukes and Smith 2000) is full of well-intentioned 
advice for how to conduct debriefing, what can go wrong, what does not work, what 
makes a good facilitator, why crews might be turned off debriefing regular flights, 
how de-identified summaries may afford learning across an airline’s workforce, and 
so forth. The recommendations are made without any associated theory of what pilots 
remember, how they remember, how remembering may be supported, or how tools and 
technology may assist in making present past events much of which has been forgotten. 
There is also no theory to explain the transition from the talk in the debriefing room to 
the actual practice of flying aircraft on line duty or in future simulator-based situations. 
The chapters in this Part B are concerned with these dimensions. The agenda for the 
remaining chapters is introduced and briefly sketched in the next section. 


LOOKING FORWARD 


The scientific and theoretical literature suggests the importance of considering the 
performative and affective aspects as essential to thinking and learning. Part B is 
devoted to these aspects of debriefing. For pilots, thinking- and acting-in-situation are 
not separate but integral to their holistic sense of flying an aircraft (Chapter 1). During 
the simulator performance, one difficult, not normally on-the-job experienced situation 
follows another. Such a context often impedes the emergence of a definite experience 
that stands out, which is the case when “no one experience has a chance to complete 
itself because something else is entered upon so speedily” (Dewey 2008). The term 
happening is more appropriate than event, for it implies the inchoate, open nature of life 
as it is experienced; the term event always implies something already finished so that 
it can be referred to as a whole. As a result of the continuously unfolding happening, 
little comes to stand out sufficiently for taking root in the pilots’ mind. Such taking root 
cannot be achieved by intellectual means alone because lived-experiencing “comes not 
by mere intellectual and outside judgment but in direct perception” (Dewey 2008). It is 
only once lived-experiencing has run its course and, because it has become an integral 
whole with a beginning and ending, that there is an object to be reflected upon. It is by 
being emotionally and practically implicated in the unfolding of the making present of 
the past in the form of “an experience” that human beings come to develop new insights. 
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The purposes of debriefing generally include more than performance assessment 
treated in Part A—though there are airlines where debriefing focuses almost exclusively 
on assessment. Because pilots get to perform in the flight simulator under conditions 
that they do not normally encounter in their everyday work, these simulator sessions 
therefore constitute the only opportunities to practice flying when things go wrong. 
That is, the simulator sessions during recurrent training and assessment also are the 
only opportunities for most pilots to encounter and learn to fly under non-normal 
conditions. The examiners have available several hundred preprogrammed types of 
problems or failures, from which the training managers of an airline select certain 
types to be used during each six-month assessment cycle. When pilots fail or do poorly 
on particular exercises, they may be asked to repeat these then and there (as seen in 
Chapter 2, depending on situational particulars, some flight examiners wait until the 
end of the session, others may immediately ask the pilots to repeat the exercise). When 
pilots fail the assessment as a whole, then they tend to be put on a retraining schedule. 
Repeated failure of assessment tasks eventually leads to dismissal. As pilot training is 
expensive, airlines are more interested in providing appropriate training and retraining 
than in dismissal—unless of course airlines only hire pilots who have paid for their 
type-rating training. Each four-hour simulator session therefore has as an important 
goal the debriefing of pilots for the purpose of analyzing what has happened and to 
communicate areas of strength and weakness. There is therefore a new order of the world 
to be created as a result of simulator training, performance assessment, and debriefing. 
This translation from represented performance to actual performance, application and 
the putting of theory into practice, is denoted by mimesis, (see Chapter 6). 

Following a description of the debriefing cultures that can be observed in aviation 
practice and how these cultures shape cognition (Chapter 5), a model is provided for the 
different orders of the world in praxis (mimesis,), theory (mimesis,), and application 
(mimesis,) (Chapter 6). The term mimesis brings to the fore that in all instances, the 
articulated order is not that of the original phenomenon but what is presented in talk is 
some form of imitation. This distinction of the different orders allows us to understand 
that Part A, whereas starting out with the order in the work of pilots (mimesis), really 
is concerned with the production of a theoretical order (mimesis,) that exists in the 
assessment scores and the verbally articulated reasons that may be found in the files of 
individual pilots and company records. The phenomenon is not how natural scientists 
produce models of the world or how sociologists and psychologists produce models 
of the social order and the human psyche. Instead, the topics of this Part B are the 
context (debriefing) and methods by means of which flight examiners, in the context 
of debriefing, constitute assessments of pilot performance. 

Debriefing has as its motive the production of descriptions and explanations of pilot 
performance for the purposes of pilot assessment and pilot professional development. 
These descriptive and explanatory texts, whether verbal or written, constitute texts 
about the world of flying. They constitute a new form of order that is derived from 
(more-or-less scientific) analysis. This order is referred to as mimesis,. It constitutes an 
order of its own, in mundane discourse referred to as “theory”. It is viewed as distinct 
even when practitioners themselves give crew resource management (CRM) workshops 
or when flight examiners use human factors-based terms to classify and assess pilot 
performances. The production of descriptions, explanations, and assessments, that is, 
the production of order denoted by mimesis,, which occurs in the course of observations 
in the simulator and in debriefing meetings, is the central topic of Part B. 
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Extensive reviews of the existing debriefing literature also focused on the 
technological aspects of the setting, including the role of multimedia. These reviews, 
however, at present limit themselves to issues such as comfort and seating arrangements 
(e.g., Fanning and Gaba 2007). The reviews have not included the function of cognition- 
relevant artifacts. Although there exist studies and meta-analyses on the role of video 
recordings and multimedia aids during debriefing for technology-enhanced simulation 
(e.g., Cheng et al. 2014), these tend to focus on measurable effects—which have been 
negligible—without investigating mitigating effects and functions arising within the 
human-human-technology transactions to make sense of collaborative work situations. 
Moreover, the studies have not sought to identify if there had been effects in areas not 
covered by the previously chosen instruments. Thus, it has been recognized that there is 
aneed for studies “investigat[ing] the circumstances in which aids such as videotaping 
may be beneficial” (Tannenbaum and Cerasoli 2013). Such research therefore should 
investigate when the use of videotaping or multimedia comes with advantages. Existing 
empirical studies, however, have not to date revealed whether and what multimedia 
aids boost—e.g., shared mental models, collective efficacy, or shared and individual 
self-awareness (Tannenbaum and Cerasoli 2013). Some issues of theoretical importance, 
including the temporal proximity of the reviewed events, still have not been investigated. 
In addition to those areas already highlighted as requiring further research, the studies 
on which the present work is based show that after a four-hour session in the full-motion 
flight simulator, the pilots often remember very little, especially very few of the details 
of the preceding scenarios and events. Even the most experienced pilots, themselves 
flight examiners for many years, may be unaware of what has happened or what the 
instrument readings had been (cf. Chapter 10). In such instances, debriefing without 
supporting tools might be inefficient precisely because the pilots are unable to make 
present again those aspects that need to be debriefed so that learning can occur. Without 
supporting devices, as the ethnographic data in airlines without a debriefing tool show, 
pilots may remain unconvinced that the flight examiners’ accounts really describe what 
had happened before. Another recent meta-analysis encourages research that investigates 
“when video should be used” and “how discussion should be structured around video” 
(Cheng et al. 2014) for the purpose of identifying the debriefing processes that bring 
versus those that do not bring about enhancement of learning. Answers to the functions 
and roles of different media, though centrally the focus of Chapter 8, are integral to all 
chapters in this second part of the present volume. 

One of the purposes of debriefing is the development of “take-home messages”, 
which, when pilots take them to heart, are designed to change their performances for 
the better and, thereby, lead to professional learning and development. In this transition 
from the world of text (in the debriefing room) that is about performance to the world 
of performance (praxis), the third form of ordering or rather reordering of the world 
occurs: mimesis,. How debriefing meetings may contribute to this transition from 
mimesis, to mimesis, is the subject of the final two chapters. 

As described in Chapter 5, pilots tend to emerge from the simulator sessions 
exhausted, having forgotten much of the details of what has happened. However, they 
sometimes have had an experience because of the extraordinary nature of an event. 
Thus, when pilots realize while flying that they have turned onto a wrong arc or have 
turned on the correct arc but in the opposite direction, they know that this will have 
considerable effect on their assessment. Apart from such experiences—i.e., completed 
units of the past that stand out—much of the inchoate stream of lived-experiencing 
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simulator sessions remains unarticulated and often forgotten. The objectives of 
debriefing includes making parts of the original situation present again—even if it 
had not been a significant event—so that it stands out to be described, analyzed, and 
assessed for the purpose of learning from practice for practice (e.g., Nergard 2015). It 
is through description, analysis, and assessment of previous experience that the pilots 
can learn from it. That is, whereas there are situations in our lives that stand out on 
their own, the purpose of the second part of this volume includes investigations of the 
instructional practices by means of which prior happenings are made present again as 
events so that they become resources for professional development. Understanding 
such instructional practices involves considering not only the role of representations, 
but also of any array of objects and performances mobilized for making present again 
what had been subject to lived-experiencing that is of a coping nature. This part of the 
book therefore presents answers to four broad questions: (a) How are memory and 
experience of and thinking in past events reconstructed in debriefing? (Chapter 7); (b) 
What is the function of tools and artifacts in debriefing generally and in making prior 
experience present again? (Chapter 8); (c) What is the role of social (flight examiner— 
pilot) relations in professional development? (Chapter 9); and How do presupposed 
underlying human factors and skills, such as situation awareness, become instructable 
and instructed matters? (Chapter 10). 
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e There is a mutually constitutive relation between the cultural practices of debriefing 


and cognition on the part of flight examiners and pilots. 


e Pilots tend to emerge from the four-hour simulator session exhausted, having 
forgotten much of what has happened. Pilots initially have little material in their 


hands to reflect upon for the purpose of learning. 
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e There exists considerable variation in the duration of debriefing meetings even 
when the same one-hour slot is available across airlines. 


e Assessment results may be announced at the beginning with or end, with 
consequences for what pilots will take away from the debriefing meeting. 


¢ To learn from inchoate, lived-experiencing in the simulator requires the past to be 
available in the present of the debriefing: it requires re-presentation. Talk constitutes 
the principal mode for making the past present again; but perceptual and bodily 
kinetic ways of presenting are important because these constitute important ways 
of pilots’ knowing. 

e Although debriefing meetings generally follow the temporal structure of the order of 
tasks in the simulator, they may be organized according to the main concepts of an 
airline’s (explicit) performance model on which assessments are based. Especially 
when a debriefing tool is available together with an explicit performance model, 
the possibility for dual determination of the meeting structure arises. 


+ Even though previous research recommends moving toward facilitated debriefing, 
few meetings are structured in this manner and flight examiner talk dominates, 
thereby mitigating how much of the pilots’ past is made present again. 


e Debriefing meetings continue to reproduce the hierarchical relations between 
examiners and pilots, and between examined pilots. Open disagreement is rare 
even when the examined pilots are flight examiners themselves who disagree with 
what their examiner tells them. 


n many fields of inquiry—including cultural psychology, cognitive anthropology, 

social psychology, and ethnomethodology (a subfield of sociology)—the irreducible 

unity/identity of culture and cognition is accepted as the starting point of all 
theorizing. This understanding is often traced to the Marxian diction concerning the 
human essence, which is recognized to exist in the specifically human, societal (cultural) 
relations that continuously produce and thereby transform society (Marx and Engels 
1978). Humans do not step into relations as if these were containers but they produce 
these societal relations with others; and, simultaneously, they are subject and subjected 
to these same relations. Even the new members to a society, from the moment there are 
born, shape and are shaped by their relations with others—rather than being socialized 
into a society (culture) which remains fixed. This insight has led social psychologists 
to the recognition (a) that “any higher psychological function was external . . . before 
becoming a function, it was the social relation between two people” and (b) that, in 
general form, “the relation between higher psychological functions was at one time 
a physical relation between people” (Vygotsky 1989, original emphasis, underline 
added). The quotation does not state that something was constructed in a relation 
and then somehow internalized or internally constructed by the developing person. 
Instead, the relationship itself eventually will exist as psychological function. A recent 
cognitive study concerned with the emergence of language provides a good example 
of this phenomenon (Hutchins and Johnson 2009). The study reports how signs emerge 
in bonobo relations and communication. Thus, bonobo mothers and infants engage in 
coordinated behaviors, whereby each party moves in particular ways to allow a mother 
to pickup and then carry the infant. Out of such relations evolves a remarkable behavior, 
whereby infants exhibit a part of their movement during pickup in a “frozen” form, 
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which serves as a gesture. In picking up the infant, the mother thereby treats this gesture 
(1.e., the frozen movement) as a request for a pickup. That is, the frozen movement 
serves as a sign in communicating to the mother the desire to be picked up. What was 
a relation first, the coordinated movement during a pickup now, in the iconic form of 
a posture, becomes a sign intentionally used by the infant. 

In human beings, the same has been shown in the emergence of the behaviors 
of individuals, which have been collective (social) behaviors first. That is, some 
psychological form that subsequently may be thought of as a (mobilization of a) 
representation initially was an aspect of joint work with others (Meshcheryakov 1979). 
An extensive example of this is featured in Chapter 9, which focuses on an instant of 
pilot learning during type rating (i.e., during the training that leads to the certification 
to fly a particular type of aircraft). Initially, that joint work is public and visible. Later, 
when the developing persons do everything on their own, this work becomes invisible. 
As a result, the behavior and underlying cognition (psychological functions) tend to be 
theorized at the level of the individual. It is remarkable, however, that this relational 
work becomes visible again when there is some form of trouble. In one study, two 
situations from very different contexts—a mother “reading” with her one-year old boy 
and a group of laboratory scientists “reading” micrographic images that have just come 
off their recording devices—are used to exhibit the identical ways in which people relate 
when reading becomes problematic (Roth 2016). Thus, for example, verbal and manual 
forms of deixis (pointing) are required for figuring out precisely where the current 
focus of reading lies and in what the object of the communicative exchange exists. 
By looking at the right location at the right time, therefore, investigators can conduct 
ethnographies of cognition (reason) and establish a cognitive anthropology of debriefing 
specifically. If researchers can report differences in cultural practices, the associated 
cognitive processes also will be different. This is particularly so when different artifacts 
and tools are used because it is precisely by means of these implements that humans 
stimulate and regulate their brain activity (Vygotsky 1989). A quintessential example is 
the knot in the handkerchief used to remind a person of something. That is, the person 
uses the physical knot to stimulate the brain into recalling some item that otherwise 
might have been irrecoverable. 

The upshot of these introductory comments is that what actually happens in 
the debriefing room is the result of the relations that those who are present produce. 
Participants both make the meetings and are subject and subjected to them. Any pilot 
learning that has occurred will have been a social relation with others. It is therefore 
crucial to understand what happens in debriefing meetings, the course of the social 
relations, because this constitutes the conditions for learning, which is the most important 
opportunity that comes with the practice of debriefing. 


CONTEXT OF DEBRIEFING MEETINGS 


The purpose of the debriefing sessions is to allow pilots to learn from what has 
happened and happened to them during the four-hour recurrent training and assessment 
sessions that immediately precede the debriefing. However, as intimated in the opening 
statements of this chapter, the examination sessions lead from one challenging task to 
another such that pilots tend forget much of the detail of what has been happening and 
leaves them with an inchoate stream of ongoing, absorbed coping (Dreyfus 1991). 
Forgetting here must not be understood in terms of bad memory. Instead, as suggested 
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in Chapter 1, much of what pilots do occurs in the form of flows. A close analysis of 
lived-experiencing has shown the need to distinguish between the perception A of an 
external object, the inner consciousness W,(A), and the object made present V¡(4) with 
the associated consciousness W(V,(A)) (Husserl 1928). Now it appears that if A was 
not present in consciousness at some time, it cannot be made present again at some 
later time in the future (Roth 2012). For example, as part of their work, pilots do not 
(and do not have to) make present again in consciousness many of the things in the 
simulator: if they need to the current indicated airspeed, they look it up. As part of the 
kinetic movements, they need to be as conscious of the actual speed value as a driver of 
a manually geared car is about the timing and speed of the clutch. But this also means 
that pilots have not made the instrument present. As a result, pilots may not remember 
their speed or the reading of an instrument (see Chapter 10). Time and again pilots 
have been observed incapable of answering their flight examiner’s question about 
the reading of a particular instrument and pertinent facts from the same flight. This 
has the consequence that in any kind of practical work context, much of the situation 
at hand is not re-presented in the mind (Heidegger 1977); this insight also has been 
taken up in artificial intelligence models that do not require internal representations on 
the part of the agents (e.g., Agre and Horswill 1997). Instead, the physical and social 
aspects of the situation in the simulator stand for themselves within the overall flow 
of the cockpit cognitive system. That is, instruments and objective conditions are not 
grasped theoretically—in terms of representations, from a removed viewpoint, and being 
thematic in circumspection—but they appear as integral parts of pilots absorbed copying 
with the ongoing task. Precisely because many aspects have presented themselves 
during the flight simulation exercises, pilots do not have equivalent representations in 
their minds. Unsurprisingly, they cannot recall much of what has happened because it 
has not existed in the form of a representation in the first place. But not all is lost. With 
some work, aspects of the preceding happenings indeed can be made present again. 

The practice of making past happenings present again for the purpose of learning 
from them is dominated by the verbal mode, which pilots and their examiners often 
characterize as “going over the head”. Pilots, having experienced the simulator sessions 
as events so packed that they are exhausted, have forgotten much of the details of what 
they had done. This situation, which arises from the regulator-driven training and 
examination practices, mitigates what they can remember and analyze for the purpose 
of learning from what has happened. There are many situations during debriefing where 
flight examiners apparently attempt to assist the pilots in recalling a particular situation 
in the simulator by asking specific questions. It is therefore not untypical to observe 
exchanges of the following kind. 


Scenario 5.1 


OL FE; You made a statement that you are supposed to use with two live 
engines but that we do not use when one engine has failed. 
> VP: Oh! Did I say acceleration altitude? 


> 03 FE: No you said climb sequence. 


Scenario 5.2 


01 FE: Do you know what your speed was when you went across after 
the rejected takeoff? 
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02 P: Yes, we rejected early. 
03 FE: When you turned into the wind. 
04 P : I was looking at the situation and I was thinking, we are getting 


slow now. So I should probably check the speed. 
05 FE: When you turned? 
> 06 P: No idea. 
07 FE: You were doing thirty-five knots. 


In the first scenario, the exchange exemplifies that pilots even forget what they 
have said in particular situations. It is apparent that the pilot responds. But what the 
pilot appears to remember is in fact something that was plausibly rather than actually 
said. This should alert investigators to the fact that some form of “filling in” occurs 
when a developing story line about what has happened is augmented by what plausibly 
might have been the case. Scenario 5.2 exemplifies that sometimes it takes several turn 
pairs—a phenomenon that tends to be called a repair sequence—before a pilot actually 
gets to respond to the apparently intended question and then fails to make present again 
the speed at which the aircraft was moving on the runway. The speed has been forgotten 
(assuming that it had been in conscious awareness at the time) even though the story 
line includes the pilot having had the idea to check speed. We do not know and must 
not speculate about what the pilot was actually doing while in the aircraft and whether 
he actually had the thought to check speed. But such having the thought of checking 
speed nicely fits this unfolding storyline that fixes the events around the rejected takeoff. 

It is apparent from Scenario 5.2 that remembering is the result of the joint work of 
pilots and flight examiners. Such exchanges make apparent the moves flight examiners 
may make to facilitate pilots’ recall of events and actions: remembering may take many 
turns at talk. Thus, for example, in the context of a specific departure, a pilot flying 
(PF) is asked about the power setting when the aircraft leveled at the top of the climb 
but fails to remember. The failure to remember continues even with further clues to 
stimulate episodic memory, such as asking about how long the power lever stayed in 
the position where it was or who eventually moved the power levers or what the torque 
reading was at the time. Pilots flying have been observed failing to remember anything 
from that part of the flight. The flight examiner then may introduce another situation 
of the particular sector flown, and, again, the pilot may not remember. The interviews 
conducted with pilots and the analysis of the videotaped debriefing meetings revealed a 
large number of instances where pilots talked about having forgotten specific aspects of 
their flight (exercises) or where the debriefing shows that they have not been consciously 
aware of their actions, instrument readings, or control settings. That is, whereas they 
were acting on particular states of instruments and moving switches and levers, which 
requires perceptions A, these were neither present in consciousness as WA). Such 
bluring of past happenings occurs especially when there are multiple scenarios flown, 
such as in spot checks, rather than in line-oriented flight training (LOFT) tasks that take 
the aircraft from some starting airport A to some final airport B. In those latter cases, a 
lot of filling in can and does occur. But even in LOFT situations, a lot of forgetting of 
specific facts about the state of the cockpit instruments and actuators can be observed, 
that is, forgetting happens even when opportunities are such that should lead pilots to 
drawing on episodic memory. Remembering apparently is facilitated when the flight 
examiners retell certain instances or when the pilots see themselves on the video (e.g., 
the video that comes with the debriefing tool). Pilots may subsequently state that they 
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have had a “flash back” and then remembered details of the flight that they appeared 
to have forgotten before. 

Interviews conducted with pilots at the end of the simulator session show that 
many of them are completely exhausted and have forgotten much of the detail of the 
events that they have undergone during the preceding four-hour training or assessment 
session. In addition to the problem of remembering under normal conditions, simulator 
sessions sometimes take place late at night and in the early morning hours. Debriefing 
following such sessions are even more difficult and pilots appear to be less capable of 
remembering what actually happened in the flight simulator. Failure to remember what 
has happened limits what can be learned from what pilots have done and undergone. 

Flight examiners know the effect of the simulator sessions on the pilots (e.g., “They 
were a bit buggered by then”) and adapt the demands on the pilots during debriefing, 
often curtailing its duration. Especially when flight examiners assessed a particular 
performance as requiring a repeat of the exercise, they debrief what happened then and 
there in the simulator followed by the repeat. Such events generally are not talked about 
again during the scheduled debriefing, though their occurrence often becomes apparent 
in references (e.g., “We talked about that in the sim”). In fact, although company policy 
may suggest the repeat to be conducted at the end of the flight simulator sessions, 
some flight examiners initiate debriefing immediately following a problematic exercise 
because they consider it more advantageous to immediately talk to pilots about the 
problem rather than waiting until the session has ended. Waiting would force pilots 
to remember something that has happened up to four hours previously. Especially in 
assessment situations (on the second day of the biannual cycle), flight examiners know 
that pilots may claim to have acted differently than they actually did. That is, during 
the debriefing meeting, pilots may claim not having acted in the way that the examiner 
describes. 

Debriefing a situation in the simulator generally circumvents the problem of 
forgetting, because the examiner can refer specifically to the issue that has just happened. 
It provides certainty about the human factor they are assessing. Thus, if the flight 
examiner were to say to the pilots, “Did you notice five minutes ago your speed was 
near the barber’s pole?” and the pilots reply in the negative, flight examiners would have 
one piece of documentary evidence that provides a better clue to situation awareness 
(see Chapter 2) than if they had to take into account the possibility of forgetting due 
to the period of time passed and the number of events that have occurred between the 
event and the debriefing. If issues covered during the debriefing in the simulator come 
up again at all during the formal debriefing, then it will be talked about rather briefly, 
to reiterate, rather than to “beat a dead horse”. A few examiners may do most of the 
major, “big-ticket items” in the simulator because, as they suggest, they can demonstrate 
a particular behavior while talking about it. Pilots tend to find this helpful, as the 
learning opportunities are made available right then and there when the situation really 
is present to them in vivid detail. The practice therefore supports cognition and learning 
in conditions characterized by high intensity and fatigue. The debriefing tool, which 
records the flight simulator sessions and makes available many instrument readings, 
may somewhat diminish the problem because the situation can be replayed in the way 
it has occurred. This allows pilots to see what they have said and done, with a clear 
representation of key instruments and actuators (see Chapter 8). 

The pilots tend to know that they had either not attended to certain aspects of the 
flight or exercises or forgotten what had happened. This mitigates the form and content 
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of their participation in the debriefing meeting: they are more likely to take the flight 
examiner’s word for what has happened, which in turn has an effect on the production of 
their institutional relation (see below). This often changes, however, when participants 
have a debriefing tool available. It is a cultural tool that serves as an external, objective 
memory and affords accessing events independently of the personal memories of pilots 
or flight examiners. Pilots often say that without the video they would struggle to identify 
exactly those steps that they had missed or to become consciously aware of what the 
other pilot was doing. Unsurprisingly, perhaps, the debriefing tool is sometimes used 
to discuss differences in the description of events that pilots and flight examiners (see 
below) provide and, therefore, it shifts the knowledge/power relations that are at work 
during the debriefing meetings. Crewmembers also often think that they have said or 
done something when, as they may see when a debriefing tool is available, they have 
said or done something different. When the debriefing tool is not available, crewmembers 
are more likely to be quiet and, thereby, contribute to reproducing the authority of the 
flight examiner over the factual matters of the simulator-based flight episodes. 

Less experienced first officers and captains tend to be more prone to forgetting 
than the more seasoned pilots, especially those who serve as training captains and 
flight examiners. The less seasoned pilots frequently are less aware of many aspects 
of the flight situations. The most seasoned pilots, often flight examiners themselves, 
generally are keenly aware of many relevant aspects of the flight situations subsequently 
discussed during debriefing. Sometimes they exhibit awareness even of less important 
detail, and they can articulate reasons why they have acted in the way they did rather 
than implementing other possible ways. But even the most senior pilots may not or 
incorrectly recall crucial aspects of flight events—such as the pilots in Chapter 10, 
who fail to notice that the remaining engine has not up-trimmed after the other has 
had a flame out. 

There are situations that crewmembers more easily remember during interviews 
and debriefing meetings. In one part of the research, the investigators interviewed pilots 
and flight examiners during the pause at the halfway pause of the simulator session, at 
its end, and following the debriefing session. Each time all participants were asked the 
question, “What stood out for you from the preceding segment of the day?” The data 
from these interviews show that when there had been “repeats”, that is, segments that 
the pilots had to fly again because they evidently had “screwed up”, then they tended 
to remember and talk about these. What has happened quite apparently was quite vivid, 
already having jelled into an experience that they could refer to as a whole and that 
therefore served as a resource for learning and improving future work situations (see 
Chapter 7). Such pilots have made some error that is significant enough to warrant the 
repeat exercise. The stopping of the simulator session, brought about by something 
that has happened, followed by the repeat of the exercise, made what had happened 
immediately before a notable, complete event. This event became a figure against 
ground, that is, against everything else that had appeared in the inchoate stream of 
lived-experiencing (see Chapters 1, 7). As an experience, a situation is lived to have 
been eventful, to have salient structure and relations, features that facilitate recollection. 
When flight examiners bring up rather minor issues during the debriefing meeting, even 
more-seasoned pilots will often say that nothing has been standing out in particular. 
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TEMPORAL FEATURES OF DEBRIEFING 


The cultural practices of debriefing differ across airlines and, with it, the forms of 
cognition involved in the debriefing event. Some of the differences that exist at a coarser 
level of description include the duration of the meetings, the amount flight examiners 
and pilot get to talk, the levels of participation, and the timing of assessment results. 
The amount of learning afforded to pilots may be anticipated to be a function of all of 
these aspects, though precise measurements still need to be conducted of how much 
pilots learn and how much their practices change as a result of debriefing meetings. 


Duration of Debriefing Sessions 


In the research in which the present book is based, all airlines (in every case regional) 
had a one-hour slot available for the debriefing sessions, including the completion of 
the paper work. If debriefing is thought of as an opportunity for learning, maximum 
usage of the time provided should be expected. Previous research suggests that line- 
oriented flight training (LOFT) debriefing sessions tend to last about 31 minutes, 
including time for watching video recordings (Dismukes et al. 2000). That research 
also suggests that longer debriefing meetings are desirable. In the present study, the 
debriefing meetings lasted 36.4 minutes on average. However, there was considerable 
(statistically significant) variance in the duration of the sessions (Roth 2015a). In 
some airlines, sessions were very short, on average taking no more than 16.5 minutes, 
essentially focusing on assessment while foregoing the possibility to create any further 
learning opportunities. In other airlines, much more of the allotted one-hour period 
in the debriefing room was used, lasting 45+ minutes involving the viewing of video 
and extensive discussions of particular events. However, even within the same airline, 
temporal outliers may be observed. Thus, in a situation involving two senior pilots, one 
of them received a 2 (poor) rating on the airline’s situation awareness scale because of 
two incorrect turns. The debriefing meeting turned out to be much shorter than normal 
(about 25 minutes), and the interviews conducted immediately afterward revealed a 
sense that there was agreement that everyone had known, and learned from, what had 
happened. If an airline relies on the debriefing tool and the debriefing tool does not 
function for one or another reason, this too may lead to the shortening of debriefing 
meeting which is premised on the opportunity to replay simulator events that had been 
marked and thereby selected for quick retrieval. The sessions tend to be longer when 
airlines use the debriefing tool due to the (a) additional time spend on actually viewing 
video recordings and (b) additional topics that arise for analyzing the behavior of the 
pilots. The amount of time spent also may be affected by the timing of the sessions. 
Thus, when the simulator session occurs in the middle of the night with a debriefing 
set for 2 a.m., flight examiners tend to keep the debriefing session shorter. 

Previous research suggests that it is important and desirable to engage in deep 
analysis of typical LOFT sessions (Dismukes et al. 2000). Yet considerable variation 
can be observed in the amount of time devoted to debriefing, which may be affected 
by the availability of a debriefing tool that provides the opportunity to review specific 
events that the flight examiner pre-selects. Previous work on aviation debriefing also 
suggests that discussions tend to be thorough when flight examiners make use of the full 
hour time slot. In the present fieldwork, more-thorough discussions were observed in 
long debriefing sessions and especially in the case of an airline that tested the feasibility 
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of two-hour debriefing sessions following three-hour simulator sessions. Even though 
the two-hour slot was never fully used, the resulting 90-minute sessions allowed for a 
lot of learning to occur. This was the case in one special situation where an airline was 
interested in finding out about possible problems that may arise in mixed-fleet flying 
(Roth et al. 2015). In that research, a total of five hours were used for debriefing two 
pilots who, after having been trained on and flown for a while the glass cockpit version 
of an aircraft, returned to the older version. That project turned up so many potential 
problem areas and sources for errors in the joint cognitive system of the cockpit that the 
airline abandoned the plan to implement mixed-fleet flying. The gist of these observation 
is this: more-extensive use of slotted debriefing time leads to more time for viewing and 
debriefing specific events, which comes with a larger number of learning opportunities. 

An increase in the amount of time devoted to debriefing in and of itself does not 
constitute sufficient guarantee for improvement; and concerns with the performance 
of a pilot may continue even when a “weak” pilot is assigned to a training pilot, who 
provides continuing support. In a number of instances, pilots continued line duty even 
though certain “issues” had been flagged during recurrent training and assessment 
sessions. Thus, even if an issue “is fixed” during another day of training, which makes 
it possible for the pilot to return and do line work under the supervision of mentors, 
deeper issues may remain. Even with ongoing supervision and training, pilots may 
continue to fail to sufficiently understand their ability. On the other hand, the very fact 
that such pilots are continuously monitored may be (intellectually) seen and affectively 
felt as a form of pressure. The pressure may itself modify pilot performance and, with 
it, any assessment of their knowledge and skill levels. 


Timing of the Assessment 


The timing of the announcement of the overall assessment differed across airlines. In 
some companies, the assessment was announced at the beginning of the debriefing 
meeting. Thus, for example, a flight examiner might open the session saying, “Okay, 
you have done well, both of you. It is a pass in both instances”, and, frequently, might 
add some form of congratulation, especially when a pilot assessed was less seasoned, 
more insecure, or has had problems before and therefore had been apprehensive. After 
signaling in this way the positive overall message, flight examiners then followed 
articulating any mitigating aspects. Thus, examiners might suggest that there were 
“some” or “quite a few” issues that manifested themselves during the simulator session 
and that therefore needed to be addressed in the debriefing meeting. 

In other airlines the result of the assessment was provided at the end of the debriefing 
session—though in many instances, such as when the flight examiners had been rating 
each exercise, the overall outcome (pass) was implicit. In one instance, a pilot flying 
had begun turning onto a 10-mile arc in the wrong direction (right rather than left), 
an error that the pilot monitoring caught in a way similar to the incorrect turn in the 
Missed Approach scenario (see box in Chapter 1). Only a little later, the same pilot had 
begun the same kind of wrong turn into a required hold. The flight examiner and the 
two pilots knew that both situations had constituted serious mistakes. The errors had 
been serious enough for the concerned pilot to think having failed the examination right 
to the end of the debriefing meeting when the examiner said that the pilot had passed 
with a low, minimum standard rating on situation awareness (score = 2). The pilot had 
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anticipated to be taken off duty and to have to enter a retraining schedule. It was only 
when the flight examiner announced that both pilots would receive their certification 
that the overall passing grade was revealed. 

It is reasonable to assume that the timing of the announcement of how well the 
pilots have performed and particularly whether they passed or failed will influence 
the pilots’ orientation to the ongoing debriefing activity. Thus, pilots worried about 
the possibility that they may have failed the examination are positioned differently 
with respect to the flight examiner talk than those who are certain to have passed and 
know that certain small problems will be addressed in the meeting. In the latter kind of 
cases, pilots may actually orient toward understanding what has happened and toward 
learning from their mistakes. In instances of the former type, pilots may fail orienting 
toward possible learning opportunities because they are consumed by worries about the 
most decisive question. That is, the practices associated with the timing of the overall 
assessment result may considerably affect cognition and learning—even though at 
present the extent of the effect still remains to be investigated. 


TALKING HEADS 


Time alone is not inherently the best indicator for the quality of debriefing sessions. 
Instead, how much is said during a debriefing meeting to account for and assess pilot 
performances may constitute a better overall indicator of meeting quality or learning 
opportunity. The basic idea is that the more individuals participate in talking about 
a topical area, that is, in using a disciplinary discourse, the better they will become 
speaking about and understanding the associated phenomena and theories (Lemke 
1990; Roth 2010). If the number of words per session is used as dependent variable, 
then there are statistically significant differences between airlines (Roth 2015a). 
Surprisingly, in that research a comparison of the number of words contributed by the 
pilots did not differ across the participant airlines. Although the mean duration of the 
debriefing sessions ranged from about 16 min to 46 min for the different airlines, the 
corresponding number of total words spoken by the two pilots together ranged from 
935 to 1,473 words. That is, across the different airlines, the pilots contributed about 
the same absolute amount of talk and, therefore, to the representation, analysis, and 
assessment of past events. There have been reports of considerable differences between 
airline cultures concerning the amount of talk devoted to crew resource management 
(CRM)-related issues versus technical issues (Dismukes et al. 2000). 

Previous work emphasizes the desirability of more pilot participation in debriefing 
and a move toward facilitated debriefing (e.g., Dismukes and Smith 2000). Regulators, 
too, emphasize crew participation—among others in the certification of flight examiners 
(CAA-NZ 2013). Thus, for example, flight examiners themselves are assessed as to 
whether they “ask open questions that determine understanding and application as well 
as knowledge”, where open question is taken to mean providing pilots with opportunities 
to talk. Flight examiners, in their recurrent assessment, have to show that they invite 
pilots to conduct self-analysis. In this study, however, the flight examiners talked 
significantly more (4,163 words) than the two pilots taken together (1,223 words). 
As shown above, the mean session duration differed from airline to airline. A better 
indicator of any difference between flight examiner and pilot contributions therefore 
is the ratio of the two, where WR = 1 means that there is an equal amount of talk of 
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flight examiners and the two pilots taken together. In the present investigation, flight 
examiners produced on average 4.3 times more talk than the two pilots facing them. 
A Bayesian test revealed that the hypothesis of a difference is over 45,000 times more 
likely than the null hypothesis, whereby flight examiners and pairs of pilots produce 
equal amount of talk during the debriefing meetings (Roth 2015a).' Differences were 
observed between airlines. In some companies, the culture was such that the word 
ratio was about WR = 1, with some individual sessions featuring more pilot than flight 
examiner talk. In other airlines, flight examiners tended to speak as much as seven 
times as much as the two pilots together, although even here, individual sessions my 
lie close to an equal talk ratio. Across airlines, therefore, the word ratios ranged from 
1 < WR < 10, over half of the values falling between 1 < WR < 4. The differences in 
the cultural practices may have been due, in part, to a shift in airline policy. Thus, in 
some airlines, changes had been recommended and were in the implementation process, 
whereby pilots were to be given more space for talking during the debriefing sessions 
than what they had been given before. 

During the fieldwork, many flight examiners commented on the difficulty of 
changing the amount of talk. They felt that the problem did not derive from them 
but from pilots, who are perceived as “wanting to be debriefed”. That is, these flight 
examiners felt pilots wanted to be more passive listeners with the load of talking shifted 
to them. Although the word ratios differed even within specific airlines, individual flight 
examiners were very consistent in the relative amount of talk they contributed to the 
meetings. Even though debriefing sessions as social phenomena are the result of the joint 
work of all participants, the word ratios of flight examiners across sessions remained 
within a small range. Thus, for example, a particular flight examiner observed across five 
sessions talked between seven to nine times more than the two pilots, whereas another 
one was observed talking between two to three times more than the two pilots. Others 
were associated with even smaller ranges of word ratios across sessions, often differing 
by no more than a small fraction of a unit. In many cases, flight examiners actually were 
not aware that they were speaking a lot more than the pilots they assess. For example, 
following a session where he articulated over seven times as many words as the two 
pilots taken together, a flight examiner still noted that the pilots “were interactive” 
and that the sessions now were more interactive than they had been in previous years. 

These results of the present investigations provide an indication of considerable 
differences between (a) an understanding of learning and assessment in a professional 
context and (b) the current ideology in formal educational setting, particularly in school 
settings. In the airline industry, the cultural practices still exhibit a stand-and-deliver 
approach, whereby learning is conceived as reception of correct knowledge from a 
designated expert. The underlying epistemology is learning as acquisition of information. 
This is so even in situations when the pilots examined include flight examiners who, 
although they may have a sense of knowing differently or better, will not speak up and 
even less contest what they are told by the individual currently serving as the examiner 
(see below). In formal educational settings, a social constructivist paradigm in one 
or another form is the commonly accepted epistemology. Within the constructivist 
epistemology, learning is theorized to occur from the active engagement of the learner 





' In traditional hypothesis testing, the probability p(data|H) is established, that is, the probability of 
the data given a hypothesis H. In Bayesian testing, on the other hand, the purpose is to establish the 
probability of a hypothesis given the data collected, that is, p(H|data). 
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rather than from passive absorption of information—especially learning in the context 
of technology (e.g., Duffy and Jonassen 1992). Other approaches consider learning in 
terms of participation in a social practice, which in the present context can be thought 
of in terms of debriefing as reflective shop talk designed to improve the work in the 
shop. Still other approaches focus on knowledge in terms of discourse competence so 
that coming to know is equivalent to learning a language. In all of these approaches, 
the instructional emphasis lies on providing learners with as many opportunities as 
possible to engage in practical work and talk. 


BODILY PRESENCE OF THE PAST 


Some research on the cognition of flying suggests that pilot knowledge exists in the form 
of (body) movement and perception sequences, rather than as knowledge somewhere 
stored in mind that is only acted out (enacted) by movements (see Chapter 1). When 
pilots communicate in the cockpit, the physical organization of the panel, their hand/ 
arm movements (gestures), and speech are integrated into a single whole “composed 
through the superimposition of several kinds of structure in the visual and auditory 
sense modalities. Granting primacy to any one of the layers of the object destroys 
the whole” (Hutchins and Palen 1997). In this regard, the notion of kinetic melody 
has been introduced to account for the bodily knowledge that is acquired from flying 
and which becomes alive in sequentially and temporally organized flows. Thus, when 
pilots discuss what to do in a particular situation—e.g., when watching scenarios where 
pilots fly under difficult conditions—they often move their hands, arms, body, and 
gaze in the way they would do if they were the pilots currently flying an aircraft (see 
Chapter 3). These movements are not merely generic but are spatially accurate, locating 
instruments and actuators (dials, switches, levers, or control column) so precisely that a 
five-centimeter change in location leads to performance errors (Roth et al. 2015). This 
accuracy therefore is similar to that between gestures and geography characteristic in the 
cognition of native peoples, where gestures exhibit great (scale) accuracy of distances 
and directions traveled. Hand/arm and body movements and orientations that produce 
again what has been done before is an important feature in the present study, where 
the aim of debriefing is to make a past flight event present in sufficient detail so that it 
may serve as a point of discussion. 

Existing studies of debriefing almost exclusively investigate talk, the number of 
words used, and the contents of the talk. This is consistent with some approaches to 
communication when it is thought of in terms of language, which constitutes the medium 
by means of which something internal to the mind is externalized and thereby made 
available to others. From this position, gestures are admitted as modes of transmitting 
information under certain conditions. To be able to reflect upon what has happened in 
the past requires aspects thereof to be present again. The presence of what has happened 
in the form of hand/arm and body movements is an important aspect of those debriefing 
sessions where pilots come to make present again previous movements. In some cases, 
these movements are exactly those that can be observed while pilots are flying. In these 
instances, there is no difference in the actual movement deployed in the aircraft and 
the movement that has representational function in the debriefing. Such work-related 
movements have been referred to as ergotic movements (gestes ergotic) that cognitively 
function as symbolic movements when they are produced for communicative purposes 
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(Roth 2003). Pilots’ knowing-how-to-fly is expressed in the movements of flying. Other 
hand/arm movements clearly are symbolic in that these stand for something else—such 
as a hand with stretched fingers used to create a visual form that stands for the aircraft 
with different bank angles (see Figure 3.5c). Some aviation-related scholars already 
recognize that communication is distributed over physical space, gestures, and words 
(Hutchins and Palen 1997). The space in which narrators find themselves, language, 
and (deictic [pointing] and iconic) gestures are deeply integrated features of cognition. 

Gestures occur frequently in debriefing meetings. There were about 34.2 flight- 
related (iconic) gestures produced per session (SD = 25.5), amounting to a little over 
1 gesture per minute given the mean duration of 31 min (Roth 2015a). The number of 
gestures produced during is correlated with the number of words (r = 0.834). A Bayesian 
analysis shows that there is decisive evidence against the null hypothesis (alternative 
hypothesis is 2,900 and 9,900 more likely, respectively). As may be expected, there 
are few gestures when the sessions are short and focus mostly on evaluation with little 
elaboration of the experienced events and how to improve upon performance. There 
were no differences between flight examiners and pilots in gesture use (about 1 gesture 
per 100 words). When debriefing sessions focused on conceptual aspects with little 
elaboration of actual flight situations, the mean number of gestures per 100 words was 
lower than in other sessions. 

As described in Chapter 1, much of the cognition involved in flying exists in 
the form of kinetic melodies such that the individual steps do not require conscious 
awareness—which is similar to the case of people who proficiently type but who are 
hard pressed if they had to state the locations of the actual letters. If this is so then 
making the past present therefore may occur when pilots are acting out when this form 
of knowing is the topic of talk. The immediate upshot is that if pilots get to contribute 
little to the debriefing, little of their knowing will actually be present for reflection. This 
aspect is illustrated in a situation where a first officer challenges the flight examiner in 
regards to the position of the hand to be held behind (pilot) rather than on top of the 
power lever (examiner). The debate centrally features the different hand positions that 
the pilot and the flight examiner display for each other as much as for themselves. In 
the situation, neither the pilot nor the flight examiner makes mere general reference to 
the two hand positions. Instead, both repeatedly position the hand relative to the body 
in a configuration precisely matching the one that had been taken by a first officer in 
the cockpit. In this instance, the first officer uses a second pointing (technically: deictic) 
gesture to indicate where the captain’s hand might be located so that his own hand can 
inhibit the other’s access to the power levers. The flight examiner, in the description 
of what ought not be done also uses the left hand to show where the hand should and 
should not be placed. This is significant, as flight examiners, in virtually all instances 
where they act out what a pilot had done, did so with hand/arm and body movements 
of that pilot. Thus, as a captain, the flight examiner would have used the right hand to 
symbolize movement of the power levers or access to the actuators on the instrument 
panels. The flight examiner then talks about and shows what the correct hand position 
is, including how it permits the thumb and pinky to actuate the go-around button. In 
a frequently observed hand/arm movement sequence, the flight examiner gestures the 
positioning of the hand and the thumb-pinky movements that modify the autopilot mode, 
which normally is followed by the power-up movement that pilots and flight examiner 
enact virtually every time they are talking about a missed approach. That is, in this 
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example, where and how the hand is to be positioned at the power lever is performed 
rather than described in words. 

Virtually all of the research on gestures relates to people gesticulating while talking 
or to how recipients understand communicative content when they can versus when they 
cannot see hand/arm gestures and body movements (e.g., Hanks 1996). However, fewer 
studies investigate hand/arm movements on the part of the recipients of a narration. 
In the debriefing meetings, flight examiners frequently narrate events accompanied 
by the relevant gestures on the part of the pilots. While listening to flight examiners 
narrating flight segments, pilots often produce the associated hand/arm and other body 
movements. In the following example, while the flight examiner is narrating back to 
the first officer that the captain had noticed the tank auxiliary pump was not on. Even 
before the flight examiner articulates the subject of the statement (i.e., state of tank 
auxiliary pump), the captain moves the right hand to where the switch and indicator 
for the auxiliary lamp are located (Figure 5.1, top panel) and then moves the hand 











| appreciate your saying one o nine, because we know 
that your altimeters are going to be good. | believe that at 
that time you noticed 





and the lights 








and then you went through the whole 
sequence together as you should in a 
two-crew situation 

















Figure 5.1 Scenario 5.3: While the flight examiner (right) narrates the event with the tank auxiliary 
pump, the pilot (left) goes through the associated movements. 
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as if pointing to the engine instrument panels of the aircraft where the caution lights 
would be on the advisory panel (Figure 5.1, bottom panel). This movement is part of 
the sequence that the flight examiner’s narrative in fact omits. In such instances, it is 
as if the pilot is reliving the original situation: the movements repeat themselves rather 
than represent some content separate from the actual movements. 

Movements such as those displayed here are integral to debriefing meetings and 
play a tremendously important role. They do not simply consist of individual gestures, 
as those exhibited in and studied in traditional gesture research, but these are movement 
(flow) sequences holding together a number of tasks (e.g., as seen in Figure 1.7). Here, 
the symbolic movement is conflated with the work-related movement. If the captain 
subsequently were confronted with the same situation during a routine flight, the 
same movements likely would be observed. Such movements therefore do not require 
representations in the traditional sense of the term because they stand for themselves. 

In the preceding example, the pilot movements appear in concert with the flight 
examiner’s narrative. In other instances, the pilot’s movements were complements to 
those of the flight examiner, featuring a different perspective on the flight. Thus, for 
example, the flight examiner might use a hand gestures to show the aircraft’s pitch when 
seen from the outside accompanied by the pilot’s hand/arm movement exhibiting the 
use of the control column (forward, backward) or the flight examiner might show a turn 
or bank angle, which would be paralleled by the pilot’s associated rotating movement 
with the control column (see Chapter 9 for a more detailed analysis of such a case). 

Flight examiners often use hand/arm and body movements to “play back” to pilots 
what they have done or should have done in the simulator. In this instance, pilots are 
confronted with a visual display that is not unlike what they see if a video recording 
is available (though front and back are reversed, because in the videotape they see 
themselves from behind whereas the flight examiner is facing them in the debriefing 
meeting). In both types of instances, the ability to correctly observe and comprehend 
a movement is tied to the ability to correctly execute the movement (see Chapter 9). 
The use of such movements leads to detailed accounts including those movements that 
are characteristic of the work practices in the cockpit. Flight examiners enact what 
should be done when examined pilots have not correctly executed some required action 
sequences. Moreover, the flight examiners’ gestures are not abstract (in the sense of 
an arbitrary relation with what they stand for) but are iconic images of those that the 
evaluated pilot had produced. That is, the flight examiners’ gestures are relative to the 
seat that the pilot had occupied during the exercise. If flight examiners were to play 
back to the captain in Scenario 5.3 (Figure 5.1) what he had done, they would also use 
a right-handed hand movement initially pointing to location 1 and then to location 2. Of 
course, one may argue that the pilots do this out of habit. But this is not (necessarily) 
the case because, when flight examiners take the position complementary to the other 
pilot during their own examinations. Thus, if the other examined pilot is a first officer, 
then the flight examiner will take the captain’s seat; and if the other pilot is a captain, 
the flight examiner will take the right seat normally taken by a pilot at the rank of first 
officer. The examined flight examiner’s movements will depend on the seat taken during 
the flight simulator session; and the examining flight examiner’s movements will reflect 
those that the examined flight examiner should have executed in the seat actually taken. 

Even flight examiners do not recall all individual items from the standard operating 
procedures specifying how to fly in a particular emergency situation (see Chapter 3). 
Instead, triggered by some event (a call, signal, environmental condition), a kinetic 
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melody—existing of connected movements (e.g., turning dials, moving levers and 
switches) and perceptions—is set offin the cockpit by some trigger and then unfolds on 
its own without requiring conscious awareness of its constituent parts. The consequence 
of this way of knowing is that pilots do not recall individual steps but instead, the kinetic 
melody as a whole has to be set off for the event to be present again. 


TOOLS 


From an anthropological perspective, human practices are cultural in nature; and human 
practices, such as those of flying an airplane, involve culture-specific tools. The same 
tools—physical or conceptual—have different functions in different cultural contexts: 
physicists and mathematicians use the same mathematical equation differently (Brown et 
al. 1989); and a design drawing of an aircraft has a different function in the accounting 
department than it has on the shop floor (Henderson 1991). Thus, tools and practices are 
interdependent and define the culture. In Chapter 1, typewriting is used as an example 
to show how movement habits come to be shaped by the physical characteristics of 
the tool (keyboard). As soon as we accept that the person rather than the brain writes 
a text, what the brain does can be seen as a function of the cultural task at hand and 
the reigning conditions (e.g., the shape of the keyboard). As apparent in the case of 
very proficient writers, who type in the manner other people speak, the thought that 
eventually appears in written form does not (always) prefigure the writing; instead, the 
thought (idea) becomes as a consequence of the writing. The writer becomes aware 
of what thinking has produced after it has come to an end. The cultural-historical 
approach, which conceives activity in terms of four levels (activity, action, operation, 
brain physiology), explicitly addresses the constitution of lower-order (brain) processes 
by high-order processes (see Chapter 1). Within that take, it is apparent that not only 
human behaviors and performances—and, therefore, human cognition—are shaped 
by the tools and artifacts that are used but also everything that takes place in the brain. 
Cultural-historical activity theory explicitly models the entire production in a singular 
unit of analysis, knowing that different tools are involved allows anticipating changes 
in the processes and outcomes. 

In the debriefing room, the available tools and representations include personal 
notes and any existing model of assessment of pilot performance, a debriefing tool 
(video), static cockpit posters, action models telling pilots what to in non-normal 
situations, onboard manuals, route and airport charts, and ad hoc representations on 
the whiteboard. Observed frequencies of use of such artifacts and representations 
are available in Table 5.1. As suggested in the preceding section, hand/arm gestures 
reproducing work-related (ergotic) movements and symbolic gestures (e.g., showing the 
attitude or bank angle of the aircraft) are present in every meeting. When airlines were 
using an explicit assessment model it was almost always part of the debriefing meeting 
in the same way as the video is used whenever available. Other representational means 
are used as the situation demands. The different artifacts, tools, and representations 
afford different ways in which prior situations are made present again, entering 
differently in the performances of kinetic melodies during the sessions (see Chapter 
8). As seen throughout this book, the debriefing situations typically involve the joint 
use of several of these tools, assembling different presentational and representational 
means into local ecologies. 
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Table 5.1 Types and Frequencies of Re/Presentational Means Used Across Debriefing Sessions. 
























































Type of Representation Frequency (%) 
Notes 

Flight examiners 97 

Pilots 69 
Cockpit Poster 38 
Video (debriefing tool) 100 
Conceptual Model 

Model 83 

Assessment Metric 28 
Action Model (F-DODAR) 50 
Manuals 10 
Instruments (debriefing tool) 94 
In-flight Manuals 28 
Charts/Maps 45 
Others (landing speed, takeoff data card) 7 
Whiteboard Drawings 38 
Hand/Arm Movements 

Ergotic Movements 100 

Symbolic Movements 100 





Note: All frequencies are adjusted to the number of sessions where the tool was actually available (e.g., 
only some airlines have an explicit human factors performance model or a debriefing tool). 


Throughout this book, the functions of different representational tools and devices 
are featured as pertinent to the context. One special device used in some airlines is the 
debriefing tool. When it is available and functions then flight examiners make use of 
it. In addition to having the capacity of replaying a video of any part of the simulator 
session, the tool also affords viewers to see actual instrument readings associated with the 
state of the aircraft at the point shown in the video (see Figure 8.1). As noted above, the 
research literature is ambivalent about the effect of viewing past performances, and clear 
evidence as to its positive effect on the improvement of praxis does not appear to exist. 
Research studies have not considered yet the role additional representational devices 
made available by the debriefing tool used in aviation. In Chapters 9 and 10 specifically, 
opportunities that arise from the use of video and the additional representational devices 
are featured, discussed, and theorized. In the light of the fact that pilots do forget much 
of what has happened, what they have said and done, what instruments showed, or their 
geographical location (e.g., see Chapter 7), the debriefing tool offers a lot of support to 
the effort of making the past present again for the purpose of analysis, understanding, 
and learning from what has happened. 


Tools Structure Debriefing Sessions 


Anthropologists have shown that cultural tools shape cultural practices and that cultural 
practices mobilize tools in culturally specific ways to any current task. This also is the 
case in debriefing sessions. Summarily stated, the available (cultural) tools provide 
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affordances for different conceptual organizations of the debriefing meetings. Debriefing 
sessions tend to be organized in the chronological manner in which the simulator 
tasks had been presented, had been recorded in the notes of the flight examiner and 
on videotape, and were completed. However, different organizations also have been 
observed: one in which a conceptual model of pilot performance shaped the organization 
of the debriefing sessions. 

Debriefing meetings often unfold in this way. During the first part of the meeting, 
flight examiners ask pilots to talk about their main impressions or to talk them through 
the examination. In their individual turns, pilots then talk about anything that appears 
or has appeared significant to them. With each pilot individually, flight examiners then 
talk about specific issues or events that the pilot identified and, through this articulation, 
make the important events and performances that affected the assessment scores. Flight 
examiners may initiate talking about certain events again during a third part of the 
meetings in the context of reviewing aspects of the day with the help of the debriefing 
tool. Some aspects of critical situations may be discussed for a fourth time when flight 
examiners ask pilots how they would rate their own performances. Finally, the critical 
situation may be sketched or referenced when flight examiners ask about the take-home 
message pilots have received. 

In the cognitive sciences, a distinction is made between episodic and semantic 
memory (Tulving 1984). Episodic memory has sensations as its information source, 
is organized in units of events and episodes, has a temporal organization, is registered 
experientially, is temporally coded directly, provides limited inferential capability, and 
is more context dependent. Semantic memory has comprehension as its source. It is 
organized into units of facts, ideas, and concepts, is atemporal, and has rich inferential 
capability. The two forms of memory are said to exist, in human individuals, side by side 
and partially overlapping. Indeed, each form is associated with different affordances. 
As shown in Chapter 1, a lot of pilot cognition is organized episodically such that when 
asked about a particular detail they perform an event sequence from the beginning 
until they get to the point to be addressed in their response—much in the way most 
humans do with notes or segments of melodies. At the same time, other aspects are 
more efficiently organized through a semantic-conceptual order, which, for example, 
allows flight examiners to talk about all the things that are of the same type (e.g., the 
different sources of power of an aircraft or all events where pilots were less aware of 
situational particulars). Although the related research suggests that episodic memory 
is more vulnerable than semantic memory, flight examiners prefer and, when they 
have the choice, opt for the chronological coverage of the preceding simulator events. 
This chronological order also characterizes the notes they took during the simulator 
sessions; and observations show that flight examiners extensively consult their notes 
during debriefing sessions. Such temporal organization was observed in most of the 
airlines. This organization may work precisely because it facilitates recall on the part 
of the pilots, who, especially in LOFT situations, retain a sense of the overall events; 
and the parts of these events are a function of the whole, leading to some “filling in”. 
In one airline observed, the explicit conceptual model of pilot performance in use 
(Mavin and Dall’Alba 2010) shaped the way in which debriefing sessions unfolded 
and in which assessment was discussed. Flight examiners may begin the meeting by 
stating that they are going to debrief according to the performance model; and in all 
instances, the laminated printed version of the visual representation of the performance 
model was on the desk between flight examiner and pilots. 


Cultural Practices of Debriefing 153 


Episodic and Chronological Order of Meeting Content 


The by far most-common organization of the debriefing meetings followed the order 
of the tasks (OCA, spot check, LOFT) in the simulator. During the simulator sessions, 
the flight examiners produced more or less copious notes. They subsequently draw on 
these notes for organizing the session. They move from the beginning to the end of 
their notes, addressing any issues previously identified and, following the simulator 
session and their constitution of the overall assessment, marked to be addressed. The 
notes function as (external) memory with sequential access, much as early computers 
with tape as storage device. Even though pilots are sometimes invited to reflect on their 
performance, which could involve any part of the preceding exercise, the chronological 
order of their occurrence and the linear order of the written notes then shapes how 
the sessions unfold and what is addressed at any point in time. The flight examiners 
describe relevant events in considerable detail and highlight, as pertinent, positive or 
negative performance aspects. Those flight examiners with access to a debriefing tool 
may include relevant video, following the same temporal order as that of the simulator 
session. This can be observed even in situations when an airline does have an explicit 
human factors model and when the flight examiners mark episodes in terms of this 
model. Thus, for example, if flight examiners observe an event such as that in the Missed 
Approach scenario (Chapter 1), they might press a button “S” associated with situation 
awareness when the pilot flying exhibits doubts about whether the go-around turn is 
to the left or right; they may also press “S” when the captain identifies the downwind 
weather situation but does not follow through with a briefing of a possible missed 
approach. Such flight examiners could have organized the debriefing meeting around 
topics arising from specific human factors; but such an organization was never observed. 

Flight examiners move through their notes from event to event until the entire 
four-hour simulator session has been covered. At that point, the final assessment of 
an often-implicit passing grade is announced or reiterated thereby ending debriefing 
followed, when relevant, by the signing of documents. The following composite vignette 
constitutes what a typical case looks like. 


Scenario 5.4 


The flight examiner initiates a particular segment: “We departed Amherst for 
Beantown and you [captain] were flying. We got airborne at 11:00. Seven minutes 
later we had a battery hot caution light and a master warning light. Now talk me 
through the situation”. Although it is principally the pilot flying’s turn to respond, the 
pilot monitoring also is involved in producing descriptions of what has happened and 
of the aircraft (battery) knowledge and the relevant standard operating procedures. 
For example, the flight examiner ascertains what the pilots know about the standby 
battery, how outside temperatures will affect the temporal runaway of the batteries. 
Then, when all relevant topics apparently have been discussed, the flight examiner 
makes an assessment-related remark: “So you made a pretty easy decision. It took 
three minutes to arrive at the decision to return to Airport A including . . .?” This 
“designedly incomplete statement” (e.g., Roth and Thom 2009) that not only allows 
for but indeed requests a pilot reply, which, then may be provided in the form of 
“Getting out and doing the checklist”. The flight examiner, in repeating the statement 
in an affirmative way, also evaluates both response and the original performance, 
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which may be augmented by saying something like, “So this was an easy decision”. 
The episodic account then continues narrating the flight in the manner it had unfolded: 
“You were out by only thirty-four nautical miles, which is not a long way out. You 
then told ATC. You made a PA. You then came back”. Queries might be directed to 
articulate in greater detail the event of coming back. Thus, the flight examiner initiates 
the talk to come by inviting pilots in this way: “What would you do differently in 
how you managed the return? If you cannot remember, we came back around the 
two hundred forty to two-hundred-fifty-knot mark until we got below ten minutes 
to touchdown. And then we were sitting at two hundred and thirty-five knots”. After 
the discussion of speed, the discussion of this LOFT goes on until every notable 
aspect has been covered. 


In this vignette, the flight examiner initiates a very specific situation, naming 
departure airport, intended arrival airport, and specific aspects of the departure such 
as starting time, the master warning light, and the amount of time that passed between 
takeoff and the warning. In the course of the vignette, subsequent aspects of the LOFT 
are noted, such as the distance from the departure airport, the communication with air 
traffic control [ATC], the completion of a PA to avert the passengers and cabin crew. At 
specific points, which in the fact that they become topics are highlighted as important, 
the flight examiner conducts inquiries, asks for elaborations, and provides evaluations 
of what pilots have said and how they have performed during the simulator exercise. 
Sometimes flight examiners explicitly formulate the possibility that the pilots have 
forgotten a relevant detail, such as the indicated air speed for a specific time period 
and the change therein at a particular stage of the flight. 


Semantic-Conceptual Order of Meeting Content 


A different organization was observed in an airline that had begun using a conceptual 
model for the assessment of pilot performance. In all sessions, the model was made 
available in laminated form on the table between pilots and flight examiners. In 
this airline, the training managers had implemented a model of assessment of pilot 
performance (Mavin and Dall’Alba 2010), associated with the ways in which flight 
examiners were asked to reorganize their debriefing sessions according to the main 
categories of the model (see Chapter 1). In the past, the debriefing practices in this airline 
had been different, following the same episodic structure described in the preceding 
subsection. The chronological approach had worked for them, as some flight examiners 
noted, because not only the pilots but also they themselves “replayed” the sessions in 
their mind using this order. Now, with the conceptual model, the flight examiners were 
asked to point to performance categories—often one or more of the “enabling skills” 
(aviation knowledge, communication, and management) before the group of “essential 
skills” (technical aspects of flying the aircraft within tolerances, decisions-making, 
and situation awareness). The following is a typical vignette representing the resulting 
debriefings in this airline. 


Scenario 5.5 


After telling the pilots that their flight-test had been successful, and stating that he 
had been quite impressed although there had been several lapses towards the end, 
the flight examiner announces to begin with the coverage of the big picture. Using 
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a circling hand gesture to refer to the image of the performance model on the table, 
pointing to and circling the knowledge category, the flight examiner starts to talk 
about it as first component of the enabling skills: So in terms of the assessment 
model here ((circling gesture over diagram of the model)) . . . knowledge ((points 
to the term)): 1 thought both of you had pretty good knowledge. Particularly the 
captain and particularly with the FMS: when the pressure was on when you had to 
reprogram the FMS, both going into Beantown and putting in the R/NAV approach at 
Beantown. As for the first officer, you went through that part of the flight flawlessly 
((circles knowledge category)). That really helped the outcome ((pen movement 
from knowledge to situation awareness)) because you knew the FMS well enough 
((circles knowledge)) that it did not cause any moments of confusion or loss of 
situation awareness or anything like that. This therefore demonstrates how a really 
good knowledge ((points to knowledge)) base helps ((points to situation awareness)) 
situation awareness, does it not? 


Here, the flight examiner articulates his assessment of both pilots as demonstrating 
“pretty good knowledge”, that is, above satisfactory performance in one of the six 
performance components of the assessment model. Rather than going through the 
LOFT in any detail, as this happens in the sessions organized chronologically and 
episodically, the flight examiner verbally gestures in the direction of two situations that 
provided documentary evidence for the “pretty good” performance in the knowledge 
category: the reprogramming of the flight management system (FMS) during one 
arrival and the use of the area navigation (R/NAV) during another arrival. The flight 
examiner particularly points to the relationship between the knowledge component, on 
the one hand, and situation awareness, on the other hand. While talking, he points to 
each of the two components, linking them in terms of a hand movement that visually 
represents the conceptual connection between the performance categories (e.g., Mavin 
and Roth 2014a). 

Riffling through their notes, where they previously marked observations using 
the first letters of the performance categories from the model (e.g., SA [situation 
awareness] or K [knowledge]), the flight examiners organizing the session according 
to the performance model then refer to other events consistent with the assessment or 
the one lapse that they may have observed in the simulator. Here, the notes serve as an 
external memory with random access typical of computers with disks. Thus, for example, 
because a pilot did not remember a relevant incident, the flight examiner described it as 
having occurred while the first officer was “running the quick reference handbook for the 
engine number one hydraulic failure”. The flight examiner, with the help of the notebook, 
named further instances that were documentary evidence of “real good” performance 
in the management category. Here, the flight examiner pointed out a relation between 
three performance aspects: during a high-workload situation, the “good” knowledge of 
the flight management system had allowed a pilot to manage different demands on time, 
with the combined result of maintaining overall situation awareness. In this manner, 
the flight examiner covered the other performance categories in the lower part of the 
diagram and how these affected those categories in the upper part. Once all categories 
were covered, the debriefing meeting was brought to its end. Flight examiners typically 
ended by asking the pilots, in one or another way, what their “takeaway message” was. 
After the pilots briefly replied to this query, the sessions stopped. 
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It is apparent that the conceptual organization leads to the articulation of bits and 
pieces from the simulator events, which tend to be decontextualized from the larger 
setting within which the issues had evolved. Such organization works against pilots 
and flight examiners, who, by and large, draw on their episodic memory for organizing 
the immediate past. This creates problematic issues because pilots forget a lot of the 
detail, which they more easily remembered (if at all) when the meeting is organized 
episodically. The problem does not lie with a conceptual organization as such, for it is 
always possible to elaborate any event that contributes to understanding the relationship 
between particular forms of behavior and the assessment category. As suggested 
above, however, the particular airline where the conceptual organization was observed 
also featured very short debriefing times. That is, any potentially existing learning 
opportunities are curtailed in the conceptual approach if it remains unaccompanied by 
detailed discussions of actual events that allow pilots to remember what they have done, 
understand and analyze their performance, and learn what they need to do to improve 
performance in the future. 


Chronological Order with Conceptual Assessment of Each Event 


Some airlines may have both a conceptual model and artifacts, such as the debriefing 
tool, that assist in debriefing episodically. Indeed, in one instance, the existing debriefing 
tool has been adapted such as to include buttons that allow flight examiners to include 
model-related marks on the simulator session record—e.g., pressing the “S” or “D” 
button when an event is associated with situation awareness or decision-making. In the 
debriefing room, flight examiners may then access events through the specific codes 
associated with them. As suggested above, the debriefing meetings nevertheless were 
organized according to the chronological order of events and the linear order of the 
notes that they had taken during the preceding simulator session. However, following 
each event, examiners then tended to make their assessments in terms of the conceptual 
model of assessing pilots’ performances. In some instances, flight examiners not only 
assessed the different components but also drew on a laminated version of the assessment 
instrument that had been developed on the basis of the performance assessment model. 
In this assessment instrument, the six main categories were differentiated into a total 
of 20 subcategories (see Chapter 3, Table 3.1). Flight examiners rated pilots on each 
of these subcategories in the way described in Chapter 3. 


Facilitated Debriefing 


Facilitated debriefing is based on the concept of student-centered learning (Dismukes 
and Smith 2000). In information processing models, declarative and procedural 
knowledge (skills) are conceived of as objects stored in long-term memory (see 
Chapter 1); learning then is equivalent to a transfer of information from the instructor to 
the learner. In contrast, pragmatic and constructivist epistemologies are based on the idea 
that humans learn by doing or by constructing abstractions from their patterned doings. 
When knowledge is concerned, this doing occurs in the form of talking; and learning 
is equivalent to learning any other language (Lemke 1990; Roth 2010). Facilitated 
debriefing is grounded in such an epistemology, where teachers are conceived of as 
guides, who, among others, assess learner needs for the purpose of creating conditions 
that meet these needs. In this approach, opportunities for learning are increased in 
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debriefing meetings when pilots actively engage in making present, talking about, 
analyzing, and critically evaluating what they have done during their LOFT sessions 
(e.g., Dismukes et al. 2000). Flight examiners act as ideal facilitators when they employ 
a set of strategies that foster a learning climate. These include (a) developing empathy, 
which involves seeing situations through the eyes of the pilots in an affective way; (b) 
actively listening and attending to what the pilots apparently want to say rather than what 
examiners want to hear; (c) eliciting information, which means restricting the amount 
of time flight examiners talk, and instead use discursive strategies that encourage pilots 
to elaborate, expand, and provide reasons; and (d) positively confronting pilots when 
their accounts appear to contain contradictions and unclear thinking, and doing so ina 
manner that does not give rise to conflict and defensiveness. 

In the present research, facilitated debriefing was infrequent—only during the late 
stages of the present investigations did one of the airlines change toward facilitated 
debriefing as a consequence of the results described here. However, in those debriefing 
sessions that were consistent with or approached facilitated debriefing ideals, the flight 
examiners invited pilots to talk about the weaker and stronger points generally or 
immediately focused the content on the chronological order of events (“How do you 
think the first segment went?’’). Thus, for example, in a session where the two pilots got 
to talk a lot (i.e., a flight examiner to pilot word ratio of WR = 1.02), the flight examiner 
invited the pilots to talk “overall” about a LOFT segment. An extended exchange 
between captain and first officer followed (740 words of the 2,004 words in the entire 
session). Some of this talk was more generic, whereas other talk was more specific 
about what had happened in the course of a LOFT. A flavor of the talk in facilitated 
debriefing is given in the following composite excerpt: 


Scenario 5.6 


In my LOFT, we were inbound. We had a fair bit of workload. At that time we also 
had that fuel low pressure. So we turned the auxiliary pump on. The low-pressure 
problem kind of went away. We then just resumed our flight path. We did not give 
much consideration to what the ramifications would be once the aircraft would be on 
the ground. I now think that we could have added these considerations in there at the 
time we were discussing the landing. So we could have chatted to the company to 
see, “Okay, is this going to be a real problem for the company if we just continue?” 
So it was not a problem for our flight. But we could have just considered that. But 
as soon as we landed I was saying, “Shit, we are stuck here”. 


The flight examiner then initiates an exchange with the pilots, contributing as 
needed to the developing account. This occurs, for example, when the two pilots are 
leaving out something relevant to the situation, such as having forgotten to “do a 7,000 
ALT select” while on a descent after a DC generator failure. Even during the facilitated 
debriefing, flight examiners point to particular segments, exercises, or situations, which 
has the effect of focusing the discussion on specific aspect of events from which the 
pilots can learn. Indeed, talking about critical micro-events leads pilots to talk about 
them, make them objects in their conscious awareness. That is, in the talk that ensues 
flight examiner intervention is observed to be again pilot-centered rather than under the 
control ofa didactically operating flight examiner. Word ratios near 1 are manifestations 
of the extensive opportunities provided to pilots for talking through topics that are either 
self- or flight examiner-generated. 
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The observations concerning the number of words spoken by flight examiners and 
pilots suggests that the debriefing cultures still operate on assumptions different from 
those on which facilitated debriefing is based. This is not to say that flight examiners 
are solely responsible, as can be seen in the realization of some that pilots act as if they 
expected to be talked at rather than (a) actively and centrally participating in talking 
through what has happened, (b) understanding why events unfolded in the way they 
did, and (c) considering what strategies they might chose for improving those aspects 
of their performances that were not as strong as desired. As any other societal activity, 
debriefing in aviation is endogenously organized and jointly produced. It therefore takes 
the collaboration and collusion of pilots and flight examiners to produce the desired 
forms of interaction (i.e., facilitated debriefing rather than talking heads). If there is 
resistance on the part of pilots to do more of the talk, then flight examiners and training 
managers will find it difficult to bring about a different debriefing meeting structure. The 
observations made in the airline that did move toward facilitated debriefing shows that 
whereas such resistance of pilots does occur, those flight examiners are more successful 
who are willing and comfortable with longer pauses, in which case the concerned pilots 
begin to actually talk and end up talking more in the session as a whole. Thus, not only 
the available tools—conceptual map of pilot performance, debriefing tool, notes taken 
during the simulator session, etc.—contribute to structuring the debriefing meeting 
but also more implicit epistemologies of learning and assumptions about what ought 
to happen during debriefing (e.g., communication of assessment versus learning from 
what has happened). 


POWER-KNOWLEDGE RELATIONS 


An important aspect of human activities exists in the different ways in which individuals 
participate. These ways usually are theorized in terms of the division of labor, a concept 
explicitly present in the model of cultural-historical activity theory as used in ergonomic 
studies of work (see Chapter 1). The division of labor also comes to be embodied in 
institutional structures; and this organization of work becomes integral part of the 
professional development of the people (i.e., subjects) in the activity system. This 
interrelation of career path and organization of work therefore is not unlike that observed 
on navy vessels (Hutchins 1995a). In the present context of aviation, institutional 
functions—such as training manager, flight examiner, training captain, captain, and first 
officer—define a hierarchy and constitute the path in the career progression of individual 
pilots. Thus, career progression tends to follow the reverse order in the preceding list, 
from first officer to training manager at the institutional level, and from first officer 
to captain and training captain at the level of the cockpit. Many studies in the social 
sciences presuppose hierarchical structures and use these to explain the structure and 
content of human interactions. Typical descriptions and explanations take this form: 
“the first officer does not speak up because of the power differences”. But the reasoning 
is circular, for the power differences (rather than an leveled situation) are derived from 
the particulars of the conversation. A different approach is to study relations and how 
these produce power-knowledge differentials (Foucault 1978). In this approach, power 
and knowledge are understood (and shown to be) tightly connected, for “there is no 
power relation without the correlative constitution of a field of knowledge, nor any 
knowledge that does not presuppose and constitute at the same time power relations”. 
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Taking such an approach allows understanding how social facts are the result of the joint 
work that people do in particular social settings. Any asymmetries of knowing, identity, 
and accountability are then understood as the product of cooperation rather than as pre- 
existing machinery in which members take specific roles (e.g., Roth and Middleton 
2006). This allows us to study power empirically, through the material relations, rather 
than having to chase some ideal construct that cannot be located. 

The foregoing take on human relations leads to the realization that it makes little 
sense to analyze systems by presupposing power and use it to explain the relations 
of participants, who are assumed to be more or less free (constrained) to act in any 
intended manner. Instead, the knowledgeable agent (subject) in/of the system, the 
objects known, and the modalities of the forms of knowledge in play are themselves 
effects of a culturally and historically constituted dynamic power-knowledge related 
work. That is, the relational structures in the debriefing room are understood to be 
the result of power-knowledge related work rather than pre-existing slots (roles) into 
which participants step and which they assume. As suggested above, debriefing is 
endogenously organized and produced; and it arises from the cooperative work of all 
participants, who, as a result, act under the very conditions that they themselves produce 
(qua subjects of activity) and to which they are subject and subjected. If, as discussed 
below, the descriptions, conceptualizations, and evaluations of flight examiners hardly 
ever come to be questioned, then this is not the result of institutional structures existing 
outside of the very human relations that make debriefing. Where would such structures 
be? And in which form would they exist? Instead, institutional structures are effects of 
joint work that unfolds such as to make operating structures visible; and these visibly 
produced structures tend to be rather stable and consistent with the image of a pre- 
supposed institutional structure. Understanding the relations within the debriefing room 
is important within a framework that takes social relations to be the first instantiations of 
all higher order psychological functions (Vygotsky 1989). As apparent from the above, 
the subjects in the activity system are effects of the relation, and so are the changes 
these subjects undergo, that is, their learning and development. 

In aviation, even in airline cultures that are more egalitarian, we observe the 
continuous production of hierarchical differentiations. As a result, pilots do not tend 
to challenge flight examiners. All participants collude to maintain the status quo. 
Specifically, (a) hierarchies are visibly produced in seating arrangements, (b) interactions 
reproduce differential power-knowledge relations, and (c) open disagreement in verbal 
exchanges is rare. All of these are visible phenomena that anyone can see who is stepping 
into a debriefing room or watching the videotapes shot therein. 


Hierarchies Among Participants 


Sociological studies have shown the close association between the world that humans 
inhabit with their bodies and their dispositions to act (Bourdieu 2000). The world 
(physically) comprehends human bodies, but these bodies (intellectually) comprehend 
the world. That is, the world is comprehensible and makes sense precisely because the 
body can represent the world outside of it thanks to its senses and brain; precisely because 
this outside world is represented in the body, it also shapes this body in a durable and 
lasting way. An example of this shaping of world and dispositions is provided above 
in the case of using a QWERTY keyboard, the analysis of which showed the match 
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between the physical structures of the world and the dispositions of the fingers and 
hands to act (Chapter 1). The keyboard is represented in the competent movements of 
the typist’s hands. That world, objectively given, not only has physical but also social 
characteristics. In fact, even the social characteristics are contingent on the materiality 
of relations. There is therefore a close association between physical space and social 
space, whereby the physical space structures the ways in which the institutional talk 
may unfold, and, therefore, the social space (e.g., Roth et al. 1996; Roth et al. 1999). 
For example, access to representational artifacts contributes to the determining who 
talks, who accesses (produces) external representations, and who shapes the content of 
the talk. In school classrooms, these artifacts may include materials for experimenting, 
seating patterns, or access to representational tools (chalkboards); and in the debriefing 
room may include the debriefing tool, whiteboard(s), and various graphical displays 
(see Chapter 8). This association also is observable in and constitutive of the cultures 
of debriefing. 

Most notably, the present fieldwork shows that with very few exceptions the 
seating patterns during the debriefing sessions invariably—and unbeknownst to the 
participants—reproduce the different institutional positions of flight examiners, captains, 
and first officers. Thus, for example, in debriefing rooms with office desks, flight 
examiners normally sit behind the office desk facing the pilots and a whiteboard behind 
them (Figure 5.2a). The same opposition between flight examiner and examined pilots 
may be observed at a round table. In some airlines, all sessions had flight examiners sit 
on one side facing the two pilots with the whiteboard behind them and a cockpit poster 
on the same wall (Figure 5.2b). In other airlines, flight examiners are positioned such 
that they sit close to the mouse controlling the debriefing tool; and the whiteboard is 
situated behind the flight examiners whereas the cockpit poster is available behind the 
two pilots (Figure 5.2c). 

Among pilots, a similar orientation to and production of institutional relations 
maybe observed, because pilots tend to be seated in the same spatial relation as they 
had been in the aircraft, with the pilot in command in virtually all cases observed was 
seated to the left (Figure 5.2). This was so even when more senior pilots—e.g., a flight 
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Figure 5.2 Three configurations of seating during debriefing meetings. There are observable differences 
in ease of access to representational devices, talk, and relations between people. However the debriefing 
room is laid out, it becomes part of the production of institution-related order and order of events. 


Cultural Practices of Debriefing 161 


examiners examined together with a captain—have been in the right-hand seat during 
the simulator session. In all the sessions observed other than type-rating sessions only 
two crews were seated with the pilot in command to the right of the other pilot. 


Relations Reproduce Differences 


The exchanges in the debriefing room were strongly hierarchical in the sense that flight 
examiners not only controlled the agendas but also generally were not questioned in 
their assessments. This was so, in general, even when the pilot pairs included flight 
examiners—including situations where the pilot to be assessed was a flight examiner 
or situated higher up in the training department than the current flight examiner. Some 
flight examiners describe the situation differently than what is observed, stating that 
they feel free, if required, to challenge and communicate what they want. However, 
in situations where the interviews revealed that examined flight examiners thought 
differently about the assessment and about what had happened in the simulator exercise, 
the evidence from the debriefing room shows that they never actually questioned their 
examining counterpart. That is, any difference of assessment that might have existed 
was never brought to the table to become the topic of discussion—for whatever reason 
and personal intention this might have been. 

In this way, there was an active production of the difference in the power over 
the assessment. In part, the situation is conditioned by the circumstance that the pilots 
are aware of their failure to remember well what has happened in the simulator. They 
know that they had been absorbedly coping with the challenges of the task. They 
know that do not remember well because of the intensity of their involvement and the 
continuous demands of the high workload. They also know that the task is different 
for the flight examiners in the observer role, which gives them an outside view on the 
events that unfold in the cockpit. Simultaneously, pilots are aware of the fact that flight 
examiners can keep track of what is happening and generally takes notes on which 
they subsequently draw. This access to the written records made during the simulator 
session provides flight examiners with access to facts and knowledge that contributes, 
to a great extent, to the production of differential power, because knowledge is power. 
Pilots sometimes suggest that they prefer undergoing the talk of the flight examiner 
rather than fighting it, hoping to shorten the session so that they can begin their well- 
earned rest. Constituted in this manner as the persons in the know, flight examiners end 
up being the authorities in the room because of the knowledge of the events in addition 
to having a vast amount of accumulated flight hours. 

In debriefing, flight examiners generally do not get challenged. They thereby have 
a hold on what constitutes fact—even when they do not necessarily recall particular 
details (accurately) and even though they may be wrong. This may be established, for 
example, in the case where the available debriefing tool shows that some situation 
was other than described by the flight examiner (see Chapter 10). There are instances, 
though, when in the course of debriefing, flight examiners find out something that they 
had not known or been unaware of. This may bring about a change in the assessment 
that had been based heretofore on their observation in the simulator. For example, a 
pilot might have been flying right turn patterns in a situation where one of the ailerons 
already had been jamming while the aircraft still was on the ground. This additional, 
aileron-related work has led the crew to be out of sequence but eventually was corrected 
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by the pilot monitoring. After reviewing the event using the debriefing tool, the flight 
examiner, initially unaware of the jamming accepts the descriptions and based on the 
pilots’ explanations, then changes the assessment. 

The different positions participants take in the division of labor of the recurrent 
training and assessment brings about differences with respect to (a) what is observed 
and (b) the degree of certainty about the recall of a fact. These differences, in turn, 
contribute to the constitution of the differences in the positioning of participants. 
Forms of positioning change, however, when the focus is on pilots as learners and 
flight examiners as facilitators. In facilitated debriefing, a shifting of authority can be 
observed toward the pilots with a resulting shift in what can be learned from experience. 


Open Disagreement is Rare 


In aviation debriefing, open disagreement is rare. Disagreement can be understood as a 
struggle for degrees of being-in-the-know, involving two or more ideas in competition 
for representing the matters of fact or matters of assessment. Like any other cultural 
field (Bourdieu 1984), the traditional debriefing room in aviation is a locale of struggle 
where the conditions and criteria of legitimate knowledgeability and legitimate hierarchy 
are determined. Denoting the phenomenon by means of the term struggle may sound 
overblown, for most of the disagreements would not have been noticeable to the non- 
initiate. Because debriefing talk does not involve overt disagreement in the form of 
statement and counter-statement, struggle realizes itself in more subtle ways. 

One of the few situations of overt differences observed is described above. In this 
case, a pilot flying is penalized for positioning the hand behind the power lever. The issue 
of where the two pilots’ hand should and should not be also arises in other debriefing 
meetings. Among others, uncertainty is apparent concerning the interpretation of what 
the standard operating procedures meant and whether an associated photograph in the 
training materials was actually correct. In the debriefing episode, the flight examiner 
insists that the pilot in front of him follow the no-hands-behind-the-power-lever 
policy currently in place. But the pilot suggests, among others, that having the hand 
behind the power level prevents the other pilot from taking over without or prior to 
announcing so. The flight examiner and pilot engage one another for over five minutes, 
each providing examples in support of their stated positions. Among others, the flight 
examiner tells the story of an experienced flight crew that crashed an airplane because 
there was neither go-around nor added power, and the pilot talks about what he has 
heard from senior company officials. Eventually, the flight examiner makes a move 
toward ending the exchange in no uncertain terms by emphatically telling the pilot not 
to deviate from the hands-on-the-power-lever policy. On this attempt to initiate the 
end of this discussion, the pilot does not abate but counters to be flying with the hand 
behind the lever. The flight examiner counters in turn, even more strongly suggesting 
to the pilot to get out of the habit of having the hand behind the power lever. Indeed, 
the suggestion is articulated in a way that it can be heard as a threat. After reiterating 
the suggestion twice, the pilot is told again to have the hand on top of the power levers 
to be able to click the go-around buttons when required. The pilot does not respond, 
which is a move to offer a concession, which thereby is part of the work that ends this 
particular struggle over the hand position. 
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In this situation, the differences in the struggle for who is in the know do not work 
themselves out initially because neither side concedes. The struggle over what is to be 
done and what the policy actually states comes to an end when the talk also includes 
the implicit threat as to who has access to and controls the assessment of the pilot. 
That is, in the struggle over being-in-the-know concerning procedure, there no longer 
is a mutual consideration of what might be done and the shortcoming or affordances 
of different ways of acting. Instead, there are repeated statements produced in ways 
that they can be heard as orders, and, because taken without further opposition, are 
taken as the current order of affairs. This only works as long as there is at least the 
perception of differential access to evaluative or punitive action, giving the upper hand 
to the participants with the (at least apparent) greater power over the other within the 
specific institutional setting. 

In Chapter 8, some of the details of an accident near Palmerston North (New 
Zealand) are provided. One of the aspects presented in more detail, and further 
expanded in a teaching sequence in Chapter 9, concerns the appropriate power lever 
position to be chosen after a missed approach is declared and the go-around procedure 
is started. As the incident report shows, there are differences of opinion in the case of 
the accident, where, because of a “hard” warning, the PF should have moved the levers 
to maximum power rather than go-around power (TAIC 1995). (In that accident, the 
flight data recorder provides no evidence of increased torque levels, though the nose 
was pitched up during the seconds between warning and collision with ground.) In the 
following scenario, a similar situation is at the base of a disagreement, which plays 
itself out more openly than normally is the case in debriefing meetings. Here, the flight 
examiner talks about a critical incidence where there is an unusual attitude (nose up). 
By asking whether the power levers were to be taking to the notch or to the ramp, the 
flight examiner can be understood as insinuating that the pilot has incorrectly acted in 
that situation. As the transcription fragment shows, the situation ends when the flight 
examiner gives in by conceding to the pilot’s reasoning. 


Scenario 5.7 


01 FE : Sowhen the nose is up, where are you supposed to take the power levers? 
02 PF : I put them in notch. 
03 FE : Are we supposed to take it to the notch or take it to the ramp? 


04 PF : The book says max power available. But the power manager was not in 
takeoff position, so I took it to the max power available. 

05 FE : So where was the power management then? 

06 PF : Power manager was in climb position. I am pretty sure about that. 


07 FE : Okay, all right. I will not argue with that. If you have had a valid reason 
for what you have done, that is fine by me. 


The exchange continues for a few turns. The pilot makes a statement about 
philosophy; there is an exchange about exceeding maximum torque and getting the 
torque needle into the red; and the flight examiner makes a final concessionary statement 
asserting that he wanted to have clarity and has not been aware of the factual matters 
as described by the pilot. 

In this situation, we observe a struggle over who is in the know about the correct 
position of the power lever. The give and take in turn 02 and turn 03 establishes that 
the power lever was put into the notch inherently evaluated by the uptake that states the 
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ramp as the desired position. In allowing itself to be heard as a question, the statement 
in turn 05 also associates itself with uncertainty about where the power lever was 
positioned prior to the action under discussion. Although the take-up itself introduces 
a hedge that modifies the certainty about the power manager to have been in the climb 
position, it is accepted as is, and thereby becomes the operating matter of fact (turn 
07). An interesting aspect of this example comes from the background against which 
the situation plays out. Thus, the acting flight examiner might be perceived to be in the 
position of power over the assessed pilot. However, this scenario depicts a situation in 
which the assessed pilot has been a flight examiner for two years whereas the acting flight 
examiner is undergoing training at the time. To complicate the matters even further, the 
acting flight examiner (in training) has about two and a half times as many accumulated 
flight hours as the examined flight examiner. This exemplifies how a priori attributions 
of knowledge or power may get the analyst in trouble, for the issue here does not appear 
to be as much an issue of relative power but an issue of objectively defining the state 
of the aircraft. Here, the acting flight examiner apparently recognizes that he did not 
attend to, and therefore could not recall, the actual position of power management. 

In both examples, we observe the participants doing the joint labor required for 
working out what the issue is and how to resolve it. It is precisely because the issues 
are articulated that the work becomes visible, whereas when everything appears to run 
smoothly the work of collusion and collaboration remains invisible. Readers should not 
let themselves be tricked into thinking that smoothly running meetings do not require 
any work and that some routine schemas—as in classical event schema theory (Anderson 
1985)—or roles—as in classical sociological role theory that conceived social actors as 
cultural dopes mechanically filling their roles (Garfinkel 1967)—make the participants 
do whatever they are doing. Thus, if disagreement is rare, then this fact points us to the 
stability of the recurrent production of a hierarchical system. As a result, pilots do not 
act in ways that could be seen as “rocking the boat”. 


CULTURE AND COGNITION IN DEBRIEFING 


This chapter is concerned with culture and cognition as phenomena in aviation debriefing 
meetings. A consistent shortcoming of all studies and meta-analyses reviewed lies in the 
fact that the outcome measures they report do not provide information on the debriefing 
as unfolding event, always adjusting itself to the contingencies of the nature of the 
simulator event, which never is the same even when the overall exercise is the same 
on paper for all crews involved. Even when different pairs of flight examiners discuss 
and evaluate the same event, as shown in Chapter 3, there are considerable differences 
in what different pairs pick out to be salient, which technical or non-technical factor 
is at issue, and which facts are pertinent to a crew’s success or failure. The following 
discussion focuses on two topics: (a) the relation of culture and cognition as it pertains 
to the structure and organization of debriefing meetings and (b) the consequences of 
the bodily nature of pilot cognition for debriefing. 


Structure and Organization of Debriefing 


This chapter, concerned with the internal dynamics of debriefing, ascertains the cognitive 
consequences of the duration of debriefing meetings, the amount of talk and nature of 
the speaker, and the debriefing culture. Thus, the duration of debriefings and number 
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of words are taken as proxy measures for the number and detail of significant events 
discussed. Because of the correlation with the number of flight related gestures, the 
duration of the meetings also delimits the amount of the original practice that can be 
made present again and, therefore, how much of flight-related knowledge is made 
present for reflection. The significant asymmetry between flight examiner and pilot talk 
shapes the relationship between (a) what is verbally described and visually articulated 
for the pilots and (b) what and how much of their bodily knowing they themselves make 
present through movement sequences. The power-knowledge differential produced 
has consequences for the assessment of the degree to which the pilots’ cognitive 
performances constitute the norm versus constituting deviations from the norm. The 
power-knowledge differential, re/produced in the seating arrangements, impedes with 
any more symmetrical approach to debriefing, such as the facilitated debrief where the 
produced conditions foster more symmetrical exchanges. 

The temporal organization of debriefing is easily supported by the linearity of the 
notes taken during the session and is associated with a great amount of apparent detail 
that becomes available as soon as the flight examiners begin to narrate and act out again 
some preceding event—a degree of detail much greater than any of the notes they have 
made. It can be observed that in those sessions organized based on major conceptual 
performance categories, the relevant details of the preceding simulator events return less 
easily and therefore do not become available to the pilots. In this type of meeting, even 
pilot actions tend to be talked about rather than performed. Questions may be raised 
about whether pilots can benefit from talk alone, for the present study also shows that 
the movement and event sequences are not propositional but associated with movement 
sequences in the form of kinetic melodies. The movements observed when pilots act 
out procedures are not symbolic representations but are the same movements that get 
the work done and orient in the work environment (i.e., ergotic movements) generally 
and in the aircraft cockpit specifically. 

The predominant order of topics was a function of the chronological order of 
the events in the simulator and the corresponding linear organization of the flight 
examiners’ notes. The order of observations recorded in the representational tool 
(notebook) drives the flight examiners’ cognitive organization of what has happened and 
how it is to be evaluated. The order also contributes to how flight examiners organize 
debriefing practice. There is no inherent necessity to discuss the simulator sessions in the 
chronological (linear) fashion in which they had unfolded. However, the notes written 
during the sessions are organized in a linear fashion, and flight examiners tend to work 
through events and associated notes from the beginning to the end. This organization 
also facilitates pilots’ recall because of the episodic nature. The debriefing tool, even 
though it affords accessing the marked events in a random-access fashion, was used in 
a way that supported episodic memory. In fact, the one time a pilot did request access 
to a specific event, the flight examiner, suggesting that this would take time, continued 
following his predetermined agenda. 

When flight examiners invited to debrief following a conceptual ordering according 
to the human factors in their company’s performance and assessment model, the practice 
was associated with the production of less-detailed accounts of what actually happened 
and with the communication of the assessment together with examples. However, the 
less-seasoned pilots and pilots in training felt that they might have gotten more out of 
the sessions if these had covered the events in greater detail. 
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One cannot talk about the debriefing sessions without also articulating the 
particulars of the contexts within which they take place. Simulator training and 
assessment are much longer than the line flights that these regional pilots conduct and, 
because the former are packed either with exercises or high-workload events, they are 
physically and affectively draining the participants. This mitigates what and how much 
pilots remember to have happened. Existing research and recommendations emphasize 
the importance of flight examiner-facilitated, crew-driven debriefing sessions (Dismukes 
and Smith 2000). However, if the pilots have problems remembering relevant detail, then 
crew-driven debriefing alone cannot be the answer to the improvement of debriefing. A 
significant amount of the debriefing sessions needs to be devoted to the work of making 
the past situation present again with sufficient detail so that any reflection on what has 
happened will be associated with pilot learning. Flight examiners have an important 
function in assisting pilots to produce the content of their previous lived-experiencing 
for the purpose of reflecting on it. This presents a challenge to a method of professional 
development through reflection on action that has not been described in other contexts 
(Schón 1987). When the cultural practices reproduce the flight examiners’ hold over facts 
and knowledge, pilots have fewer opportunities to make present what has happened. 
But if pilots cannot make present what they have done, they also cannot “learn from 
experience”, which is the quintessential purpose of having debriefing in the first place. 
The failure to make present past events mitigates shared cognition in the room and what 
pilots may take away as learning outcome, that is, in the changes that the debriefing 
meeting can bring about and thereby how it can affect pilots’ cognition in future events. 

In the fieldwork on which the present work is based, considerable variation was 
apparent in the relative amount of talk that the flight examiners and crewmembers 
contributed to the debriefing meetings. Existing research suggests the desirability of 
increase in the amount of crewmember talk (Dismukes et al. 2000). However, even 
in the more egalitarian culture where the research took place pilots tended be more 
passive in the constitution of what has happened, definitely mitigated by their awareness 
that they have forgotten much of the detail that the flight examiners then provides. 
This determines what they want from the debriefing session and shapes their form of 
participation. Especially less-seasoned pilots find it more useful when flight examiners 
narrate back to them what they did (wrong), tell them why they have been wrong (“the 
theory”), and make recommendations as to how to improve. In the context of current 
learning theories, however, what the less-seasoned pilots would like to see happen 
may not actually lead to the desired learning. This is because pilots are not proactive. 
Indeed, they currently find it less useful to take a more proactive role and debrief 
the simulator session on their own. This then works well with the existing situation 
where many flight examiners talk a lot. But talking a lot does not mean pilots take 
away more than if they participated to a greater extent. Although the flight examiners 
do most of the talk in many of the sessions, the reactions on the part of the pilots are 
not necessarily negative. Indeed, some of the sessions with little pilot talk (WR > 6) 
receive very positive comments and the pilots have few if any suggestions for how to 
improve the debriefing generally and the sessions relating to them specifically. Some 
pilots, on the other hand, again especially the less-seasoned ones, not only appreciate 
detailed discussion and representation of important events but also, when detail is not 
provided, suggest the provision of such to improve debriefing sessions. The awareness 
of having forgotten also contributes to the relevant authority over just what the factual 
performance had been, and, therefore, how it was to be evaluated. 
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The Value of Articulating Bodily Cognition 


Debriefing is useful only if the pilots actually remember what they have done, recognize 
how what they have done varies from best practice, and actively work to change their 
actions. If pilots were not remembering what has happened, they would not be learning 
from their experience. They would not have to go into the simulator but might just 
listen to any flight examiner or flight instructor lecture. Because much of the cognition 
in an aircraft cockpit is situated and distributed (see Chapter 1), the presence of past 
experience likely is shaped by the extent to which pilots and flight examiners can and do 
act out what has happened. The present fieldwork revealed a direct correlation between 
the amount of talk and the number of flight-related gestures and body movements. 
Flight examiners who use the debriefing tool know that it assists pilots in seeing and 
understanding what they have done; when flight examiners present past events in talk 
alone, pilots have the sense that things are going over the top of their heads. But it 
is not only the video that can lead pilots to feel they are learning. Pilots tend to have 
similar feelings when the debriefing allows them to articulate in detail what they have 
experienced; because pilots forget, this articulation may require a lot of joint work with 
the flight examiners (see the extensive description and analysis of such situations in 
Chapter 7). Performing and talking through flow sequences (kinetic melodies) makes 
present again, for everyone to see, what the pilots have done or what they should have 
done. In contrast, when the flight examiners simply describe events verbally, pilots may 
not know what they are talking about, leading to what some call the “foggy look in 
pilots’ eyes”. Seeing themselves and performing situations allows pilots “to understand 
what the flight examiner is trying to tell them in their own mind’s eye”. 

Flight-related hand, arm, and body movements produced in the debriefing room 
serve symbolic functions. The shape of linguistic signs is arbitrary with respect to the 
shape of the phenomenon that they denote. Iconic signs bear a perceptual similarity but 
tend to differ in terms of the physical dimensions of the designated objects. However, 
the movements that pilots and flight examiners produce for the purpose of symbolizing 
actions and perceptions tend to be highly accurate when compared with the production 
of these movements in the actual cockpit. Thus, an analyst familiar with the aircraft 
may easily identify what someone is talking about even when the soundtrack is turned 
off because the movements are not merely generic but reflect the physical layout of the 
cockpit. For example, in a go-around procedure, the sequence includes Flap 15, gear up, 
check the power followed by setting heading (HDG) and indicated air speed (IAS) on 
the flight guidance control panel (Table 1.1). There is a characteristic hand movement 
that goes with this sequence, where the pilot monitoring moves the flap control levers 
to 15, moves the landing gear handles into the up position, and a hand movement 
towards the torque gauges, which is followed by a hand movement upward to the flight 
guidance control panel to select HDG and IAS (Figure 1.7). It is a characteristic overall 
flow or bodily kinetic melody (see Chapter 1). This flow is not encoded in terms of its 
part—e.g., as separate positions and movements for flaps, gear, instrument check, and 
flight guidance control panel settings—but unfolds on its own once triggered. 

In anticipation of Chapters 7-10 concerning the function of movement sequences, 
the following general statement concerning gestures are advanced. We understand 
these movements in terms of cultural-historical psychology, where any higher 
order psychological function was a relation first in and with the social and material 
environment. That is, what later appears to be ability exclusively situated in the mind 
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was a relation first prior to being something attributable to individual cognition. The 
ability retains this quality such that when required, one can observe a “renewed division 
into two of what had been fused in one” (Vygotsky 1989, original emphases). The 
movements observed occur as if the pilots were actually sitting in the cockpit (simulator 
or line flight) only that the material environment is absent. These observations are 
consistent with long-term cognitive research that documents the genesis of symbolic 
gestures in work-related movements (e.g., Roth 2003). Observations and recordings 
during line flights and in the simulator show that the symbolic movements are identical 
to the flying-related movements. That is, what appear to be symbolic gestures and 
representations are in fact presentations, the same movements doing different kind of 
work in the two situations. This is not always the case, as in some instances, always 
involving (less-seasoned) first officers, pilots can be seen to walk to a cockpit poster to 
act out movement sequences; these sequences are relative to the artifacts now in view 
and present. That is, the presence of the representation has produced a configuration 
that supports the acting out of kinetic melodies, which these pilots did not and perhaps 
could not do while seated in the cockpit. Flight examiners also can be observed modeling 
movement sequences in front of and relative to the cockpit posters—especially while 
working with less experienced pilots (especially those in the type-rating phase). Or 
flight examiners might ask a pilot to show something on the cockpit poster or the 
instrument panel presented by the debriefing tool. In Chapter 8, these observations are 
further elaborated through detailed analyses of debriefing cases involving the various 
representational devices available in the debriefing room. 
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KEY POINTS 


+ Debriefing is a cultural-historical activity, which, when conceived as a unit, has 


all the characteristics of society as a whole. Work is required not only to produce 
any outcomes from the debriefing meetings but also to produce and maintain the 
conditions of the debriefing meeting itself. 


Humans do not represent in their mind much of their everyday life activity. They 
walk but do not place feet; and they talk without choosing and retrieving words 
from long-term memory. This places challenges on recalling “experience”, for 
only those aspects of life can be represented that had been (consciously) present 
originally. Understanding debriefing therefore requires dealing with the differences 
between lived-experiencing, creating an experience, representing an experience, 
and reflecting on an experience, all of which requires re/presentation. 


Mimesis is the imitation or representation of the order of the world. The order 
of the world made and exhibited in everyday practice constitutes a first form of 
mimesis (mimesis,). In talking about the world of practice, a second form of order 
is created referred to as mimesis ,. When human factors researchers talk and write 
about practice, mimesis, is equivalent to scientific theory, whereas practitioners’ talk 
about practice tends to be commonsense theory (“folk models”). Finally, mimesis, 
refers to the reordering of practices produced in the movement from theory back 
into praxis. 
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light examiner: Walking into this debriefing meeting, I did not know 
whether I was going to pass that pilot or not. But as we talked, as soon 
as I came to understand what the pilot had been thinking during that 
approach, it was quite obvious to me that I would not be able to give a pass. 


Pilots are assessed repeatedly each year for the purposes of (a) ascertaining their 
proficiency and (b) providing training to fly in dangerous situations not normally 
encountered on the job. A normal session includes a one-hour briefing, a four-hour 
period of flying ina full motion simulator, and a one-hour debriefing. In the introductory 
quotation, the flight examiner begins by repeating what was said immediately after 
having observed a crew for four hours: the uncertainty about passing or failing the first 
officer. That is, even though the flight examiner has had four hours to observe pilot’s 
performance, uncertainty remained about whether the pilot could pass or should enter a 
retraining program. The flight examiner expresses uncertainty even though a number of 
different things characteristic of the culture were done (see Chapter 2) to ascertain the 
problem existed in the pilot rather than somewhere else in the cockpit joint cognitive 
system. Because flight examiners also have to make recommendations as to subsequent 
forms of training—either on the job, line training, or off the line during retraining and 
re-examination—finding out the root causes of performance issues is important. Any 
such recommendations will be used to determine the areas and the levels of training 
that are required. These root causes are to be articulated by means of an assessment 
discourse, which, in many airlines contains a set of specific categories—including 
situation awareness, management, decision-making, and knowledge (e.g., of standard 
operating procedures)—assembled into some form of pilot performance model with or 
without graphical representation. The assessment discourse and how it is established is 
the focus of the first part of this volume. In Part B, the focus is on cognition, technology, 
and work of debriefing. The interesting dimension of debriefing is that (a) it is itself 
a form of human (social) activity requiring cognition and (b) its products include the 
assessment of the assessed pilots’ cognition, especially those aspects of their cognition 
concerned with flying and as manifested in relevant human factors terms. 

In the second part of the introductory quotation, the flight examiner states that once 
the thinking and thought processes were understood, it was clear that the pilot had failed 
the examination. That is between the instant of going into the debriefing room, when 
the flight examiner did not know whether the pilot would pass or fail, and the instant 
in the debriefing room when the assessment was announced to the pilot, the meeting 
produced as its outcome whatever was required for the decision. What did the meeting 
produce to arrive at that point? How did the debriefing meeting unfold to arrive at the 
stated conclusion? How did representational devices change the meeting to bring out 
what had happened and what the pilot was thinking? These and associated questions 
are answered in Chapters 7 through 10. In this chapter, a foundation for understanding 
the cognitive work required to understand pilot cognition and to produce conditions 
for their learning. 

Flight examiners note (as one of the mitigating factors of debriefing work) that 
pilots often do not remember what they have done. This leads to further questions. 
What is the joint work of remembering required for making the simulator experience 
a basis for learning? What is the joint work that flight examiners gloss as “getting into 
the head of pilots” and “getting their trains of thought”? Answers to these questions 
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are important given the fact that despite the considerable use of debriefing techniques, 
as meta-analyses show, the peer-reviewed literature on debriefing practices actually is 
minimal and does not provide answers (Fanning and Gaba 2007). Whereas meta-analyses 
do identify positive effects on individual and team performance, there continues to exist 
a need to “further refine our understanding of why [debriefings] work and how best to 
deploy them” (Tannenbaum and Cerasoli 2013). On the other hand, there are also many 
studies in which debriefing has not led to increased accident prevention; and there may 
even be negative outcomes (Gist et al. 1999). 

Most important perhaps is the need for some theory of debriefing. It does not 
suffice to conduct empirical research that shows learning differences between teams 
that have been versus those that have not been debriefed. Such research provides no 
information whatsoever about the cognitive processes at work by means of which 
aspects of lived-experiencing come to be re-presented in the form of narratives, how 
such narratives are analyzed, and how learning accrues such that the subsequent work- 
related performances of the agents actually improve. This chapter lays the theoretical 
foundations for understanding debriefing as cultural-historical activity and the cognitive 
work involved in going from the inner consciousness during absorbed coping at work to 
its grasp in the form of an experience. This chapter is also about the different orders of 
the world, the order apparent in praxis, theory, and reconfiguration of praxis by theory. 


DEBRIEFING AS CULTURAL-HISTORICALLY SITUATED ACTIVITY 


Chapter | introduces cultural-historical activity theory as a framework for understanding 
any productive human activity that contributes to the generalized satisfaction of needs 
that exist in society as a whole. This theory distinguishes itself from all others currently 
available in making the entire productive activity, from beginning to the finished 
product, the appropriate unit of analysis for understanding human cognition. The theory 
has found adherents in the ergonomics community, for it explicitly brings together 
cognition, the work-related tools and artifacts, and the work process as a whole. It 
also comes with a particular take on the form of practices and cognition. Thus, in any 
particular investigation the unit of analysis has to be chosen such that it contains all the 
characteristics of society as a whole. This is so because of the progressive differentiation 
of society through repeated divisions of labor: earlier forms of activity split up and 
give rise to two or more forms of activity that accomplish what the original activity 
has done but generally at a new level of complexity. As a result, the later appearing 
activities have characteristics that are already contained as possibilities in earlier 
appearing forms of activity. 

The motive of the activity includes and is shaped by the uses that the product will 
have; that is, any activity is conceived of not for itself but for its role in society. This 
is so also with debriefing, which has its origin in the human capacity to look back at 
an experience for the purpose of looking forward to make future experiences more 
positive and performances more effective. That is, the outcome of debriefing activity 
leads to better and safer performances in the activity of commercial transport operation. 
Whereas practitioners have looked at what they have done to learn from it for thousands 
of years, the appearance of facilitators and consultants to aid in making sense of past 
performances is more recent. This is the historical dimension of the activity. The cultural 
dimension also is apparent, because debriefing does not occur as a regular feature of 
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work in many other professional cultures. The addition of third parties, where it does 
occur, tends to come in the form of specially trained personnel (e.g., psychologists, 
psychotherapists in debriefing oriented to minimizing post-traumatic stress) or others 
who have distinguished themselves through their outstanding professional capacities 
(e.g., experienced personnel in hospital training settings or flight examiners in aviation). 

Cultural-historical activity theory presents a dynamic, change-oriented model that 
makes thematic subjects, object, tools, outcomes, rules, community, and division of 
labor (Figure 6.1). In debriefing, the working objects include observations, records, 
and whatever else pilots and flight examiners recall. That is, the preceding form of 
engagement, flying in the simulator for the purpose of training and assessment, now 
becomes the object/motive of the current activity. It becomes the object not in the way 
it has existed before but refracted through whatever is recovered, articulated in the form 
of representation, and made present again (e.g., through video or through narratives 
produced in the meeting). In the course of the debriefing activity, these materials are 
transformed to produce descriptive accounts, conceptual (explanatory) accounts, 
assessment, and instruction and plans for improvement. Inherent in the theory is change 
that occurs in any part of the systemic whole. Thus, the participants are changed while 
doing debriefing. Of interest are those changes that are pertinent to the improvement 
of flying performances, though there may well be other changes as well. Indeed, 
some of the most apparent changes include the additional exertion of the participants, 
who have already spent five hours in the facility—one hour briefing and four hours 
flying—and now are spending another hour debriefing what has happened so far. Other 
apparent changes pilots may undergo include awareness of what has happened before 
and learning what they may not have been aware of before (e.g., Chapters 7 and 10). 
Some of the changes may actually be counterproductive, as this has been documented 
in the case of indigenous midwives, who learned in UNESCO workshops particular 
ways of talking about their work without parallel changes in what they were actually 
doing (Jordan 1989). 

As any other activity, debriefing is not just a box or machinery into which the 
participants step and where they play particular roles. Instead, participants themselves 
make the debriefing meeting and exhibit to each other everything required such that 
they or any witness would recognize what they are doing as a debriefing meeting. That 
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Figure 6.1 Debriefing as conceived in cultural-historical activity theory. The theory makes explicit the 
role and function of tools and artifacts in the production of outcomes, which include descriptive and 
explanatory accounts that may enter the assessment report and the records of the pilots. 
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this fact often goes unnoticed on the part of researchers interested in debriefing arises 
from the fact that the debriefing meeting apparently is oriented toward the production 
of accounts, discussion of events, or construction of explanations using human factors 
discourse. One aspect of that work of talk exists in the different ways in which speakers 
modalize what they are saying. For example, adding “I think”, “I suppose, “probably”, 
“T guess”, or “may be” turns a definitive statement into something that is possibly only 
partially correct, represents a particular viewpoint, or is simply false. A pilot might 
say, for example, “I probably should have stopped, listened and then completed the 
checklist”, which does different conversational work than if the statement had been “I 
should have stopped, listened, and then completed the checklist”. By modalizing the 
statements, the speakers organize the meeting, for example, by setting up a context so 
that they do not lose face in the case flight examiners treat the statement content as 
incorrect, partially correct, or inappropriate in some other way. Speaking in this way is 
part of the joint work that makes debriefing meetings unfold in the special ways in which 
they do. By saying, “It is okay to say no” after pilots reply affirmatively when asked 
whether they have learned something, flight examiners explicitly contribute to the work 
of managing the conversation itself rather than strictly talking about aviation issues. 

Much of the work that produces the debriefing context within which the different 
social actors act, however, remains invisible. It is precisely when this work unfolds 
unproblematically that it remains invisible (Suchman 1995). The observation that 
disagreements are rare and that debriefing meetings unfold in recognizable ways 
normally without trouble (Chapter 5) is a manifestation of the underlying collusion 
that brings about and maintains the conditions for continuing to debrief in the way 
that it currently occurs and that it normally is known to occur. Whenever someone 
formulates what is presently going on, then the organizing work becomes explicit. Thus, 
when pilots give two different answers to the same question, even if difference is as 
little as that between “stand by” and “stand by flight director”, flight examiners might 
formulate it as “confusion”. Flight examiners might say to pilots, “I am not trying to 
set you up!” and thereby provide a frame for how to hear what they have said or what 
they are going to say. 

Formulating the current event is much more explicit when speakers verbalize 
what they are subsequently doing, such as when flight examiners, working on the 
assessment records say they are “going to save that at the moment and finish it later, 
tomorrow”, “P1l just wait for this file to close so I can log out, leave, and shall return 
in five minutes”, or “I'll just go out of the room for five minutes and Il have you two 
discuss the first two exercises”. In these cases, descriptions of the ongoing event are 
projectively oriented, articulating something like a plan for what is going or supposed 
to happen. Such statements then are integral parts of debriefing meetings explicitly 
oriented towards the organization of the meeting. But in being work, they also require 
the associated cognition. 


EXPERIENCE, COGNITION, AND LEARNING 


Any theory of debriefing needs to account for the ways in which three different points 
in time are related. First, pilots are caught up in, act in, and co-produce events of 
which they are consciously aware only to some extent. That they do not need to be 
consciously aware is so because expertise does not require conscious (theoretical) 


174 Cognition, Assessment and Debriefing in Aviation 


attention but amounts to being attuned. Moreover, real praxis always has an open 
future. The result is that our lived-experiencing is and remains inchoate, because life 
never stops. Life also does not come with markers of beginnings and endings so that 
we never know until some form of closure is achieved that there is some definite event 
or temporal period that stands out from the stream of life. Thus, for example, the pilots 
of TransAsia Flight GE 235 (see Chapter 10) did not know that they were in the middle 
of a disaster. That whatever was happening was part of a disastrous crash event can be 
said only with hindsight. This then was a moment of definite closure. That is, there are 
indeed moments of closure: out of the stream of lived-experiencing something like an 
experience comes to stand out. Second, in the debriefing meeting, pilots have to make 
present again what has happened and aspects of their past performance. But this presence 
of the past cannot be other than through representation or renewed bodily presentation 
(as in “flash backs”). Third, learning from experience requires analysis, a form of work 
that tends to be made thematic by the term reflection on practice. 


Lived-Experiencing versus An Experience 


In Chapter 1, experience is presented as a category that captures the relation of persons 
and their world; a description is provided how this is relationship is intellectually, 
pragmatically, and emotionally reflected in the person. In thinking about experience 
in this sense it is useful to distinguish between sensual, lived-experiencing (Ger. 
Erleben and Erlebnis) and an experience as something accomplished (Ger. Erfahrung). 
Experience as it is defined in Chapter 1 pertains to the former. The sensual lived- 
experiencing is characterized by its indivisible durational nature, its continual flow 
associated with a flow of consciousness (Schiitz 1932). As a result, “what we live in 
duration is not a being, not something discrete and well-defined, but a steady transition 
from a now-and-thus to a new now-and-thus”. Inner and outer aspects are irreducible 
parts of a sensuous whole open toward the future that is characteristic of a situation 
(rather than a person). A distinction can be made between (a) lived-experiencing, which 
inherently is incomplete and open to the future (i.e., inchoate) and (b) experiences as 
things that are gone over in mind after these have ended and therefore can be referred 
to as an entity. An experience, because of the theoretical nature of talk about something 
in which it is rendered, is an intellectual phenomenon. A contrast thereby is introduced 
between (a) sensuous, lived-experiencing that also is affective because of its purposive 
and volitional nature and (b) purely verbal accounts of (represented) experiential 
wholes. The contents of such accounts are supersensual. The original sensuous lived- 
experiencing is important to what it means to know, “for without it [the thinker] would 
never know what it is really to think and would be completely at a loss in distinguishing 
real thought from the spurious article” (Dewey 2008). In actual practice, there are trains 
of ideas, “which form a train only because they are much more than what an analytic 
psychology calls ideas. They are phases, emotionally and practically distinguished, ofa 
developing underlying quality” (Dewey 2008). A distinction is drawn between ways of 
theorizing thinking-in-action as it flows and the decomposition of thinking into purely 
intellectual qualities that nowadays are referred to as representations or thoughts. This 
distinction is important here where it is shown that debriefing sessions in aviation 
include not only external representations (or inscriptions) but also presentations, that is, 
ways of making aspects of a situation present again in modes that are (nearly) identical 
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to the ways in which they appeared in the original lived-experiencing (cf. Chapter 8). 
With respect to tools (A), for example, the distinction is between those that are simply 
present and ready-to-hand, with the tendency to be absent from conscious awareness, 
and those that are also made present internally and therefore present-at-hand (V(A)) 
(cf. Heidegger 1977). The difference is between immediate relations to the world in the 
sense of ecological psychology and an access to the world by means of representations. 

The feel of the two forms—lived-experiencing while being absorbed in sensuous 
activity, which is in principle unreflected (Schiitz 1932) and an experience manifested in 
representations—is different. For example, in the debriefing sessions, pilots might look 
ata flight instrument or read a checklist from their quick reference manual (QRH), which 
requires the same underlying (physiological) processes as during flight; or they might 
go through a sequence of hand/arm movements symbolizing actions in the cockpit and 
the movements are identical to the actual hand/arm movements required to push buttons, 
throw switches, or move levers. The dials, switches, or hand movements do not need to 
be represented while debriefing a flight segment; these are ready-to-hand in the same 
way as they are when part of the lived-experiencing while flying the segment. Thus, in 
everyday practice we are caught up, coping with the world while accomplishing any 
relevant present task. Things are happening, “but they are neither definitely included 
nor decisively excluded; we drift. We yield according to external pressure, or evade 
and compromise” (Dewey 2008). That is, we are not detached from the world, merely 
object over and against other objects; instead, we are caught up in the flow of life 
and constitute a flow. There thus are no externalities; and there is no side-by-side of 
independent things in themselves (Schiitz 1932). In our sensuous lived-experiencing 
there are changes but there are no genuine conclusions. Thing replaces thing, “but does 
not absorb it and carry it on”; there is lived-experiencing, “but so slack and discursive 
that it is not an experience” (Dewey 2008). 

In the actual practice of everyday work, the premises of actions become apparent 
only when the conclusion manifests itself. What is theorized to occur for the lived- 
experiencing of thinking also is valid for the lived-experiencing of practical action. For 
pilots, thinking- and acting-in-situation are not separate but integral to their integral 
absorbed coping where now-and-thus transforms into now-and-thus while lingering only 
long enough for pilots to have a sense of continuity. During the simulator performance, 
one difficult situation follows another. Such a context impedes the emergence of a defined 
experiential whole because “no one experience has a chance to complete itself because 
something else is entered upon so speedily” (Dewey 2008). We may have to add here 
again that Dewey did not have available in his language the differentiation German 
philosophers, sociologists, and psychologists have at their hands between sensuous 
lived-experiencing of the here-and-now and experience as something finalized that can 
stand out. When the sensuous lived-experiencing does not have a chance to finalize 
itself as an object (i.e., as an experience), it does not really deserve this name because 
there is little to nothing that takes root in the minds of the pilots. 

In the study of debriefing, one has to be concerned with another phenomenon. 
Because pilots come out of simulator sessions having forgotten much of what has 
happened, the challenge for the joint work of debriefing is to make parts of the original 
situation present again so that something actually is available to be described, analyzed, 
and assessed; or sufficient material has to exist so that the descriptions allow pilots to 
remember and construct a sufficient detail for reflection and learning to become possible. 
It is only when it is described that an experience can be analyzed for the purpose of 
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future learning (see below). That is, whereas Dewey describes those situations in our 
lives that stand out on their own, this study is concerned with making what was lived- 
experiencing present again so that it becomes an experience. 

Perception plays an integral role in Dewey’s theory of experience, a concept that he 
distinguishes from recognition. In bare recognition, there is no development occurring 
and active perception required in receiving something new is stopped before something 
can be learned. Perception constitutes an act of reconstructive doing, which leads to the 
coming alive and renewed becoming of consciousness. This active perception “involves 
the cooperation of motor elements even though they remain implicit and do not become 
overt, as well as cooperation of all funded ideas that may serve to complete the new 
picture that is forming” (Dewey 2008). Recognition alone is insufficient to lead to vivid 
conscious awareness: lived-experiencing is vivid without having to be reflective. The 
involvement of the whole person is important because we cannot understand thinking 
and acting by taking intellect plus affect. In lived-experiencing, thinking and emoting 
mutually pervade each other. In (vital) sensuous lived-experiencing, the intellectual, 
practical, and emotional cannot be teased apart and are parts of a single whole (Schiitz 
1932). Indeed, the emotional, intellectual, and practical aspects ofa situation are different 
manifestations of an experiential whole in the way this is theorized in Chapter 1. 

Recent work on cognition of flying emphasizes that the appropriate unit neither is 
the individual pilot nor the interaction between the pilots; instead, the appropriate unit 
of cognition is the cockpit as a whole, that is, the cockpit as a joint cognitive system 
(Chapter 1). This aligns itself with a pragmatic and cultural-historical view of lived- 
experiencing as involving the environment, the person, and emotion. Thus, “lived- 
experiencing [perezivanie] is that unit where, in an irreducible form, is represented, on 
the one hand, the environment—lived-experiencing always refers to something that is 
outside of the person—and, on the other hand, how I experience it” (Vygotskij 2001, 
original emphasis, underline added).! The individual pilot therefore does not just act 
but is caught up in the constitutive relations that make the cockpit the unit of cognitive 
analysis. When the individual pilot reflects on what previously has happened, one obtains 
the retrospective view of one part of the joint cognitive system on the system as a whole. 
Writing about the viewer of an artistic piece, the notion of active perception is important 
because it refers us to the fact that “a beholder must create his own experience. And 
his creation must include relations comparable to those which the original producer 
underwent” (Dewey 2008). In this act of perception, something significant is extracted, 
and an abstraction takes place. This perspective is pertinent here, because during the 
debriefing sessions pilots also come to look at their past performance through the 
lens of a special spectator, the one who more easily than others can include relations 
that they themselves had undergone. The upshot of this way of thinking about lived- 
experiencing is that for pilots to learn from their simulator sessions it is not sufficient 
to intellectualize it, understand it by means of representations in the classical sense 
of the term. Instead, if there are conceptual representations used then these have to be 
directly linked to the concrete lived-experiencing that is happening while being part 
of the cockpit joint cognitive system. 





' In his native Russian, the psychologist Vygotsky used the term perezivanie to translate the German 
Erlebnis and Erleben (cf. van der Veer and Valsiner 1994; Zavershneva 2016) as distinguished from 
Erfahrung (Rus. opyt) both of which are rendered in English as experience. 
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Every constituted experience—that is, an experience (Ger. Erfahrung) rather 
than the sensuous lived-experiencing of being then-and-there in the midst of all (Ger. 
Erlebnis)—s either impression or reproduction; as reproduction it is a making something 
present or not (Husserl 1928). However, past events are not present again in the way 
they originally existed. Access to past events may be represented diagrammatically to 
theorize how what is present today may shape future experiences. An original, once- 
occurrent and never re-accessible Event E, is seen at some later time ¢, as El, at time £, 
as E?, and at time E, as (Figure 6.2). The past event is present (again), contemporaneous 
with the present, but generally not in its original form. Instead, it is adumbrated by 
everything that has been happening since. Thus, for example, at time ¢,, the original 
event is seen through the lens of event E, at the time, that is, through £?. At any time 
tin the debriefing room, the lived-experiencing is marked by the immediate presence 
of other people and things. The past is also present in adumbrated form (see vertical 
line downward from £,), where some of the intervening layers between the original 
experience Æ, that is to be discussed and the experience of the immediate presence £. 
Perhaps because of the intensity and temporality of the flight simulator sessions, the 
pilots tend to forget much of what has occurred; but if for some reason some aspects 
of E, jell into an experience and stand out, then the event tends to remain vivid—as 
happened in the case of the repeatedly mentioned pilot who had taken two wrong turns 
(see Chapter 5). Right to the end of the debriefing session, when told the license was 
renewed, this pilot believed to have failed the examination. The repeated incorrect turn 
may in fact eclipse everything else that might have stood out from the preceding flow, 
and even eclipse what was following (e.g., £). 

In the case of debriefing, not all aspects of the original event are accessed in this 
way. Some of the aspects present in the original situation, such as an aviation chart, a 
quick reference manual, or the movement required to set a switch or operate a lever, 
are present again in the same mode rather than in the form of a representation. Their 





chart, >» chart, 

quick reference quick reference 

handbook, handbook, 

body movement body movement 
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Figure 6.2 In this diagram, the horizontal line represents a series of now moments (1) and associated 
living, sensuous experiences (£,). At a future time ¢,, experience Æ, is present not in its original form 
but as Ey, seen through all other forms of experience that have occurred since then. Here, Ey is present 
as seen through £” and Ey, that is, through the lenses of £, and E, as these appear at time ¢, constituting 
experience £. 
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contents therefore can be consulted in the way they had been consulted during the 
earlier flight event. As a result, they are then immediately available for access and 
practical use at time ¢, and as part of E, in the way that they had been used at 1, as part 
of the earlier, now mythical Event £, (Figure 6.2). That is, in this case, aspects of the 
past are immediately present and do not have to be represented—in other words, they 
may be said to represent themselves. The immediate presence of artifacts from the 
original event, in contributing to the reconstitution of the factual aspects of that event, 
facilitates the processes of remembering (e.g., Middleton and Brown 2005). It has been 
shown that even when someone else provides a description of what has happened, 
apparently forgotten events are felt as “coming back” (Roth 2017). The story that is 
currently elaborated and developing facilitates remembering events that have not been 
memorable on their own. 


Presentation and Representation 


Important to understanding cognition in debriefing is the fact that this practice requires 
the presence of the past. That is, in whatever suitable manner, the past—whatever has 
happened in the flight simulator—has to be made present so that participants can engage 
in its analysis for the purpose of assessment and learning. The predominant approach 
to cognition has its origin in the work of the German constructivist philosopher I. Kant 
who suggested that we know and act in the world through inner, mental representations. 
In this approach, (mental) representations are the conditions for having any experience. 
Psychologists have come to distinguish between internal representations and external 
representations (e.g., diagrams, graphs, maps, and images). In some social sciences, 
the latter have been theorized under the heading of inscriptions and associated social 
practices (Roth and McGinn 1998). In that approach, researchers investigate what 
people do with transcriptions, how they transform, accumulate, transport, and deploy 
them before seeking recourse to any mental stuff in their explanations. 

There is an ongoing debate, however, whether representations are indeed 
necessary—some artificial intelligence scholars pursue approaches that show how 
intelligence is possible without reason as conceived in the traditional way and without 
representations (Brooks 1995). Modeling studies show that artificial neural networks can 
be trained to recognize nouns, verbs, and objects without having explicit representations 
thereof (Elman 1993). Moreover, cognitive scientists have shown that if Tetris players 
would have to use representations to interpret what is displayed on the computer screen 
and to plan their future actions, they would act an order of magnitude slower than they 
actually do (Kirsh and Maglio 1994). When pilots do have to interpret events, such 
as when an unusual signal occurs marking an non-normal flight event, they tend to 
be slow—as shown by the reaction time between a ground warning proximity system 
warning “Terrain, Whoop Whoop Pull-Up, Whoop Whoop Pull-Up” warning and the 
initiation of a pull-up maneuver, which simulations show to be of the order of 5.4 
seconds (TAIC 1995). Artificial intelligence researchers have shown that short-order 
cooks can prepare something like ten orders simultaneously, which would be a near 
impossible feat if they had to have an internal representation of all orders, as this is 
required in the classical approach to situation awareness (see Chapter 1). Short-order 
cooks achieve this level of performance because they transform states in and of the 
world rather than because they process mental representations of the kitchen and its 
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contents (Agre and Horswill 1997). Thus, in most mundane situations, human actors 
do not fashion representations but are intimately connected with an into the world. 
There have been suggestions that the mind maintains a set of deictic pointers to relevant 
objects, processes, and states in the world rather than representing states of the world 
(Ballard et al. 1997). Presence of structured things in the world, handiness, and direct 
connection with the world appear to account better for everyday performance and human 
experience than models based on internal representation. In most normal, non-failure 
(breakdown) situations behavior thus is better modeled psychologically in terms of 
absorbed coping rather than in terms of mental representation (Agre 1997). 

In debriefing meetings, participants do the work required to make previous events 
present again. Because the participants cooperate producing these meetings with respect 
to form and content, which requires them to make available to one another whatever 
is needed to succeed, they provide the cognitive anthropologist with the data in which 
cognitive and practice reveal themselves. In their efforts, participants in debriefing 
employ a variety of means. What is the function of these means, which include tools 
and artifacts? There are increasing numbers of studies suggesting that complex socio- 
technical systems, such as nuclear power plant control rooms, navy vessel bridges, or 
aircraft cockpit, should be considered as joint cognitive systems where the information 
passing between parts should be core to the analysis, theory, and cognitive engineering 
perspective rather than the mental content (alone) (see Chapter 1). As a result, such 
research points to the importance of studying interactions between agents—e.g., by 
studying the flow of inscriptions (external representations)—rather than looking at what 
an individual might think or represent. To understand such a system, “it is not enough 
to understand the properties of the individual cognition or even of individual decision 
making with decision aids (though such knowledge may certainly be helpful)” (Hutchins 
and Palen 1997). This joint cognitive systems view is consistent with the definition of 
lived-experiencing as a phenomenon that includes people, (their intellectual, affective, 
and practical characteristics), the material and social environment, transactional 
relations (mutual determination), and emotions (which are private only in pathological 
cases). Individual lived-experiencing is only a one-sided reflection of collective lived- 
experiencing, itself part of the flow of life. 

One important upshot of this discussion is this: we may anticipate the process 
and content of the debriefing meetings to change if aspects of the cockpit as a whole 
are present or represented versus when the focus is only on what individual pilots can 
recall (see Chapter 8). Although reviews of debriefing have included considerations 
of the setting, these considerations were limited to such issues as comfort and seating 
arrangements (e.g., Fanning and Gaba 2007) and did not include the role of cognitively 
relevant artifacts. If performance and aptitude cannot be reduced to the individual pilot 
or to the interaction of pairs of pilot, and if the cockpit as a whole is the appropriate 
unit of analysis, then the quality of the debriefing sessions might improve when parts 
of the original joint cognitive system comes to be re/presented in its holistic nature. 
This might facilitate the process of making the original situation present again for the 
purpose of analyzing it to yield higher levels of future performance. While evaluating the 
performance of their peers, even flight examiners will often enact the same movements 
that they would use in the cockpit to articulate what the pilots in a videotaped scenario 
should have done (see Figure 3.6 and Chapter 5). We may anticipate that relevant 
artifacts—representations of instruments, cockpit layout, and flight-relevant artifacts 
such as maps or manuals—increase the likelihood of making the situation present again 
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because there is already something of it present. Therefore, artifacts and representations 
of the original situation can be anticipated to change the quality of the reflective 
processes from which changes in pilot performance may arise. 

Gestures constitute an important aspect of communication across cultures and even 
in the blind community; and, as seen in Chapter 5, they also constitute an important 
communicative means in the debriefing room. Gestures are integral not only to the 
ways in which people communicate but also to spatial perception and orientation 
and other cognitive capacities. Gesture and speech together produce multilayered 
representations important to understand the decision-making process of the cockpit 
joint cognitive system. Symbolic hand/arm movements often arise from work-related 
(ergotic) or information-producing/seeking (epistemic) hand/arm movements (Hutchins 
and Johnson 2009; Roth 2003). In such instances, the original work-related movements 
are present again at a future time (e.g., Henry 2000) and therefore function much like 
the cockpit artifacts that are brought into the debriefing room. The movements are not 
represented in the ordinary use of the term, that is, by means of something else that may 
(in the iconic case) or may not (in the abstract case) have perceptual similarity with the 
content of the representation. The hand/arm movements are not a representation but 
presentations. The distinction between present and past thereby comes to be blurred. 

The making present of past events tends to involve, besides words and gestures, a 
variety of tools, artifacts, and signs. Thus, humans may affect their mind by means of 
external sign/tools, for example, enhancing memory tasks by means of the proverbial 
knot in the handkerchief. Whereas remembering requires some form of representation 
of the past, recognition may occur in those cases where something like a movement 
is experienced again. This experience thus is not represented but present in the same 
way that tools may be present to hand without an accompanying representation of the 
tools in the mind of their users. 


Cognition in and Reflection on Praxis 


The distinction of theory from praxis, the actual sensuous doing of praxis without 
timeout, is a common aspect of the everyday world. In my own research, I found teachers 
complaining that what they have learned at the university, theory, has taught them very 
little about what they actually have to do when they enter the classroom. Similarly, 
during their classroom-based instruction electricians are asked to learn trigonometry 
to be able to calculate where and by how much to bend electrical conduit; but in their 
everyday work on the job site, they never use trigonometry but bend conduit using the 
marks of their conduit bending tools. During fieldwork in the aviation industry, similar 
talk can be heard. Thus, some pilots complain about the talking-heads approach during 
their traditional CRM courses, and less than one half of the pilots in one airline actually 
found the didactic mode of traditional classroom-based instruction useful (Mavin 
and Roth 2015). Similarly, although the notion of joint cognitive systems has a lot of 
currency in the academic world, training managers and airline consultants note that all 
this talk makes little or no sense because in their everyday praxis, they have to train 
and assess individual pilots. In the same way, the concept of situation awareness is so 
prevalent in the context of debriefing and CRM training that any critique of the notion 
is met as academic nonsense (i.e., mere theory) that has no grounding in the actual 
work of training and assessing pilots. Good theoretical concepts often are cast aside as 
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mere “academic jargon” (Mavin et al. 2013). But even in their everyday work, pilots 
sometimes have a sense of the encountering talk that makes little sense in the context 
of their work, such as when new human factors-related discourses come to be adopted 
and implemented by the training department. 

The fundamental idea of underlying debriefing is that pilots may learn from 
reflecting about their preceding performances, that is, from reflecting on the praxis of 
flying. There exists a considerable body of research on the role of reflection pertaining 
to a variety of fields, including nursing, policing, occupational therapy, education, 
and aviation (e.g., Mavin and Roth 2014b). Distinctions also are made between 
“reflection-in-practice” and “reflection-on-practice” (Schón 1983). But much of the 
research does not consider that all reflection requires re-presentation of a past event 
and, therefore, involves a considerable change in cognition that can be predicted based 
on cultural-historical activity theory (see Chapter 1, Figure 1.9). Reflection-in-action 
is a contradiction in terms, as the phenomenon requires a stop-and-think approach that 
makes present the past to become the object of thinking (analysis). This is so because 
in aviation, flying an aircraft and reflecting on flying an aircraft are two very different 
phenomena, requiring different forms of cognition because the objects of cognition 
differ; this distinction is apparent when the two activities of flying an aircraft and 
debriefing the flying are compared (i.e., Figure 1.9 versus Figure 6.1). That is, the 
cognitive aspects of reflection are hardly discussed, especially the difference between 
cognition in the practice, here flying, and the cognition of reflection, which in the 
current context occurs in debriefing. Understanding how pilots may learn in and through 
debriefing, therefore, requires attention to the differences between the practice of flying 
and the practice of debriefing. Different approaches to understanding the cognition 
of flying an aircraft are presented in Chapter 1. Here, a brief outline of cognition in 
debriefing is provided, a line of inquiry continued and deepened in the course of all 
subsequent chapters, especially in Chapter 7. 

The cognition of flying an aircraft can be understood in terms of kinetic melodies 
that do not require the mental representation in conscious awareness of actions or action 
schemas (see Chapter 1). From the perspective of the individual pilot, much of what 
is required in flying is present-to-hand such that they do not have to recall to mind or 
represent the required aspect because, like the different instruments, they are present and 
therefore can be directly perceived. Consider again the analogy with typewriting. The 
typist does not need to recall from memory the positions of the keys on the QWERTY 
keyboard; and the typist does not need to make present the schemas for moving the hands 
and fingers. Instead, the competent typist proceeds in the same way as most people speak: 
the content forms and develops without explicit representation in the consciousness of 
the writer. The presence of the keyboard suffices for the sentences to take shape on the 
monitor. The essence of knowing to typewrite thus exists in the actual relation between 
typist and keyboard. We may also say that the essence of knowing to type exists in the 
praxis of typewriting. Knowing to fly an aircraft also exists in the irreducible unity/ 
identity of pilot and environment, including the other crewmember(s), the physical 
layout of the cockpit, and the functional systems behind instruments and actuators. 

When the pilots and the flight examiner are in the debriefing room to talk about 
what they have done, for the purpose of reflection-on-action, then those events that 
previously occurred in the flight simulator have to be made present again—unless 
they are present in the situation, such as when pilots and examiners bring the onboard 
manuals, flight charts, and so on. Any reflection has an object, which, in the case of 


182 Cognition, Assessment and Debriefing in Aviation 


reflection-on-practice, is past practice. For reflection to occur at all, therefore, the past 
event has to be made present so that it can become the object; in other words, reflection 
requires representation. Any theory of debriefing, where reflection on and assessment 
of practice is to occur, therefore requires attending to representation, remembering, 
and memory. Only when there is an object can the process of reflection unfold; that is, 
only when the past event is present in some form can any organization of past events 
occur from which something may be learned to inform future events. The presence of 
the past requires some form of technology, which, for the most part, exists in the form 
of language (discourse), but in the case of aviation, also includes different artifacts and 
devices (e.g., the debriefing tool) that have been present in the past event and therefore 
contribute to stimulating episodic memory and to the process of making present again 
the event (see Chapter 8). These technologies constitute the tools by means of which 
humans regulate the activity of their brains, and through the brain the activities of their 
bodies (Vygotsky 1989). The construction ofa narrative that occurs during the retelling 
constitutes exteriorization in the public space by means of material marks, inscriptions, 
and discourse (language); and such exteriorization not only makes possible but also 
necessitates the structuring of primary understanding by explanation (Ricceur 1991). 
The narrative of a previous flight event and everything else that goes with it may then 
be analyzed and interpreted. It is precisely the interpretation of such narratives that 
“culminate in the self-interpretation of a subject who thenceforth understands himself 
better, understands himself differently, or simply begins to understand himself”. 

The construction ofa narrative in which the past becomes present again is not value 
or theory free. It depends on the technology of representation. The medium itself is the 
message (McLuhan 1995) and indeed constitutes an ideology (Volosinov 1930). The 
verbal aspect of the narrative of the events in the simulator then constitutes a form of 
life history, the making of a piece of life story, a form of fiction—which, deriving from 
the Latin fingére, to form, shape, has the same etymological root as the word “fact”. The 
result, to be intelligible at all, is and has to be subject to the same requirements as the 
historiographic and novelistic styles characteristic of biography and autobiography. As 
the story of the past simulator-based event is told, an emplotment of characters (pilots, 
air traffic control, passengers, and cabin crew), things, and environment occurs. That 
is, as the event is told, “the narrative configuration contributes to modeling the identity 
of the protagonists of the action as it molds the contours of the action itself” (Ricoeur 
2004). Any theory of action is tied to a theory of representation. In debriefing, however, 
we are not dealing with internal representation of past events in the minds of pilots but 
with their constitution of the object of reflection in a public forum accessible to all. A 
theory of action, therefore, can be modeled on the theory of text because any action 
or thing, to be the object of analysis, is (and has to be) represented in and through the 
narrative produced in the course of debriefing. 

Debriefing, a form of praxis where reflection-on-action occurs, constitutes a 
junction between past and future praxis. To understand how debriefing might change 
praxis—i.e., how the praxis of debriefing changes what pilots do, and, therefore, 
cognition in the course of flying—requires some theory about the relation between 
the order of the world as it appears during flying, the order of that world as rendered 
during debriefing, and the relation between the two orders in subsequent flying. This 
articulation makes apparent three different orders: praxis that has happened in the past, 
theory and other talk about praxis, and application, that is, the movement from talk about 
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practice (theory, CRM training) back into the praxis of flying. These three orders may 
be theorized in terms of three forms of mimesis, a term that pertains to the imitation of 
the practical world in and through communication, most often talk. 


PRAXIS, THEORY, APPLICATION: THREE FORMS OF MIMESIS 


We may distinguish three different kinds of orders associated with the distinction 
between praxis, theory, and application. The first kind of order is that which practitioners 
produce and exhibit to one another while doing their work (mimesis, ). A second kind of 
order exists in the form of theory or talk about work praxis (mimesis,). These two orders 
frequently are juxtaposed and made thematic in the discourse about a theory—praxis 
gap. But it is well known that these gaps remain even when practitioners talk about 
their own praxis (Bourdieu 1980); thus, practitioners’ talk about practice is of the same 
kind as theoreticians’ talk. Finally order or rather reordering is occurring when talk 
about praxis (theory) is taken back into praxis and thereby reshapes it (mimesis,). In this 
framework, mimesis, takes an intermediary position by taking us from the world we have 
lived in to a future world transfigured by the configurations of mimesis, (Ricceur 1984). 
This then allows investigators to ask questions about how the concrete world comes 
to be configured and reconfigured by talking about and theorizing it; in other words, it 
affords investigating the relation between the prefiguration of the practical field and its 
refiguration through the practical application that are the result of the intermediate step. 


Praxis and Mimesis, 


First, there is the practical, mundane world of work, referred to as praxis when the 
emphasis is on the actual, sensuous doing of the work and on the inchoate nature of 
absorbed coping and the associated lived-experiencing. When the pilot monitoring 
and the pilot flying in the fatal TransAsia GE 235 (see Chapter 10) have the following 
conversation, they exhibit to each other the current states of affair (from Aviation 
Safety Council 2015). 


Scenario 6.1 


01 PM: Okay, engine flame out check 
02 PF : Check 
03 PM: Check up trim yes, auto feather yes 


As this fragment shows, as an integral part of their very work, the two pilots 
articulate its order. The pilot flying, in saying “check” (turn 02) also marks that he is 
checking—unless, unverified and unverifiable here, the talk is just talk contrary to the 
common and assumed practice that “check” means that the check has been or currently 
is conducted. Similarly, in turn 03, the PM formulates that he is checking up trim and 
then provides the result of the check, “yes”, which, in more elaborated terms, may 
be glossed as, “I checked up-trim and, yes, the engine is up-trimmed”. The verb “to 
formulate” is used here to indicate that practitioners do not just talk and do not just act, 
but in their talk and action articulate for the other what they are currently doing. Such 
formulations may be even further removed from the action when, still continuing to 
fly, pilots provide each other with a more removed account of what has just happened. 
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Thus, for example, the PF in the Missed Approach scenario (see Chapter 1), after the 
initial parts of the go-around procedure, says something like: “I’m sorry about that I 
thought it was a right hand there for a second”. He thereby explains, as part of their 
praxis of flying, why what has just happened was a little bit disorganized and flown 
in a non-standard manner. Having thought that the turn is to the right required the PM 
to correct the PF, which entailed that the PM’s workload immediately increased and 
the entire go-around sequence was assembled in a way inconsistent with the standard 
operating procedures. During the TransAsia flight, something of the same ilk happened. 
Thus, the PF stated shortly before crashing, “wow pulled back the wrong side throttle” 
(Aviation Safety Council 2015). He thereby accounted for what heretofore could have 
been the beginning of an account that may be glossed in this way: “We are in this 
situation because I pulled back the wrong throttle, that is, the throttle of the live engine”. 
Mimesis, refers to all these ways in which practitioners articulate and thereby exhibit 
for one another the orderliness of the world as they currently perceive and witness it 
and without the benefit of hindsight. They explicitly make available the apparent order: 
to other participants and to themselves. 

Any constraints on the ways in which this accounting work in living praxis can be 
done arise in and from praxis itself. This talk has to make sense in that situation, where 
anything actually available in the field does not have to be represented in the mind 
because it is accessible as context of the talk when and where required. Thus, whereas 
the context for the pilot talk in Scenario 6.1 would have to be articulated in the world 
of text (e.g., in a novel or in a research article), it goes without saying for the three 
pilots who are present in the cockpit at the time. The interlocutors are present to each 
other and to the situation and to the circumstances of their talk. The talk immediately 
makes sense, where sense is understood not only in terms of what the talk is about (the 
Anglo-Saxon “meaning”, the referential function of talk), but also in five additional 
ways that are even more pertinent to Scenario 6.1: (a) this talk as part of a system of 
talking (pilots recognize the words as part of their language); (b) talk as a deliberate act 
of communication on the part of the speaker; (c) talking as a form of acting that is the 
foundation of talking as an act of communication; (d) talking as a form of positioning 
the recipient, who thereby is actively solicited to reply; and (e) the here-and-now of 
this talk as related to an encompassing in-order-to motive (Schiitz 1932). 

Talk in the cockpit makes sense because it is integral part of the different contextures 
that come with the surrounding environment, among others, by means of deixis and 
ostension. Deixis may operate by means of a pointing finger and hand configuration or 
by a head and body turn in the direction of the thing being talked about. It also operates 
at the level of language, such as when the pilot in the Missed Approach scenario says 
“sorry about that”, where “that” refers to the immediately preceding events generally and 
to the instant when he had forgotten the correct direction of the turn more specifically. 
Other frequently used indexical include “this”, “there”, “here”, and “it”. But even 
when the pilot flying in the Missed Approach scenario says something like, “It doesn’t 
look crash-hot out there in the east”, not only “it” and “out there” are indexical, but so 
is “crash-hot”. This is so because what the term “means”—or, more accurately, what 
the pilot monitoring can find out about what the pilot flying means to say without 
saying so—can be found out by looking outside the window into the easterly direction. 
Whatever the PM finds by looking is “it”, and this “it” is to be seen as “not crash-hot”. 
Knowing one’s way around the pilots’ world and knowing aviation language are so 
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interlaced here that the two become indistinguishable. That is, the order of the world 
and mimesis,, in many instances, are part of the same phenomenon. This is why at the 
very end of his life, the psychologist Vygotsky realized that the external field and the 
field of sense were not two different and opposed phenomena but that the two were 
indeed two manifestations of one and the same indivisible phenomenon (e.g., Klochko 
2010; Zavershneva 2010). Newcomers to the field of aviation may hear pilots talking 
but “will not understand a word” even if they know and are familiar with every word 
that the pilots articulate; the newcomers will not understand a word because they do 
not know the world of piloting and, therefore, lack any way of making out just what 
the pilots are talking about. 


Theory (Talk about Praxis) and Mimesis, 


In debriefing, there is more-or-less principled talk about the world. Everyday ways of 
talking about past events tend to be less principled, though requiring a representation of 
the everyday world in the same way that completely theoretical accounts (e.g., human 
factors) require. Theoretical accounts of the world will be shot through with sets of 
interconnected concepts, a fact that is at the origin of the often-decried theory—practice 
gap (Roth et al. 2014a). Thus, when flight examiners make use of a human factors model, 
their assessment talk will reflect the model. This is so in part because the assessment has 
to be defensible, which it is when it adheres to the standards chosen by the airline; to 
another part, the assessment discourse will be the ground for any remedial training, which 
pilots have to undergo when their performances have fallen below the company-defined 
minimum standards. Thus, flight examiners might list a number of observational facts: 
the power levers had not been moved up when the torque was sitting at 0% for a while; 
the speed remained too low risking to lead to a stall of the engine; the two pilots were 
talking about taking off their headsets; and the pilot had been distracted in other ways 
while leaving the critical factor of speed unattended. Following this narrative, flight 
examiners might then provide a theoretical rationale and assessment by articulating a 
series of human factors and their relations. Such a rationale might take the following 
form: “On the evaluation form, situation awareness is marked. Because your situation 
awareness was down, decision-making was not adapted to the situation: you simply 
stepped through your procedures regardless of the circumstances. You have failed to 
adapt the procedures in the course of your task management’. 

In this instance, a self-sufficient text would have been articulated, including both a 
descriptive account of the flight event as observed and a theoretical account in terms of 
human factors. Thus, the fact that the pilot was unaware of the critical speed situation, 
which could have let to a stall, is attributed to situation awareness, one of the human 
factors that appears in the airline’s performance and assessment model. In this reasoning, 
the decision-making was not adapted to the situation because situation awareness was 
down. Instead of changing around the order of the standard operating procedures so 
that these take into account the current conditions, the pilot is said to have implemented 
them literally; flight examiners thereby co-articulate that a situation may require the 
deviation from the order inherent in the standard operating procedures. On the other 
hand, appropriate task management could have assisted in this situation, for example, 
by eliminating or delaying the talk about the headsets. The assessment statement thereby 
highlights that task management would have allowed the pilot to adapt the procedures. 
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But the evaluation could have simply stated that appropriate task management would 
have improved awareness and, thus, freed up resources for appropriate decision-making 
regarding the adaptation of procedures. 

In this situation, the descriptive account (what has happened) and the theoretical 
account go together. They are part of a textual world removed from the actual praxis of 
flying. This is so because in the pilot’s record, the rationale for a particular rating (e.g., 
management = 2 [poor]) has to be justified by even a briefest of empirical evidence, 
which exists in the form of a behavioral description. The categorization of the problem 
under “management” entails particular recommendations for actions that the pilot is 
asked to take; and it entails actions to be taken on the part of the training captain (or 
captains) assigned to assist the pilot in improving performance and for the purpose of 
passing the next routine assessment three or six months down the road. Even further 
removed from the actual praxis of flying are theoretical discourses and models, such as 
the “Model for Assessing Pilots’ Performance” (Mavin et al. 2013) used by a number of 
airlines in the Tasman region. Even though pilots may be trained in using the theoretical 
model that had been abstracted from flight examiner talk, pilots may find it hard to 
understand, even harder to apply to concrete aviation scenarios, and almost impossible 
to translate into concrete actions to take for improving performance. Thus, it comes 
without much surprise that these researchers report considerable variations when pilots 
rate their peers shown in a video scenario. Even though there were 92 pilots participating 
in the study, the standard deviation on any one of the six human factors involved was 
a full point on a five-point scale (e.g., something of the order of decision-making 
DM = 2.5, SD m7 1.0). In other words, given that a little over 68% of scores lie within 
+1 standard deviation, this finding shows that about 32% of the participants attributed 
ratings that were more than | unit away from the mean. 

The textual world of talk about and theory of praxis, though produced in locations 
away from praxis, is not completely detached from and without reference to it. This 
is so because the reading and interpretation of text—e.g., when the training manager 
reads an assessment and the associated descriptions and rationales—require a mundane 
understanding of how the world works. Thus, for example, reading the final report of 
the Air Bagan W9011 crash on December 25, 2012 (see box in Chapter 7) presupposes 
familiarity with the world of aviation. Take the following excerpt from the crash report 
section entitled “Situational awareness”. It states: 





While on final approach at an altitude of 660 ft, the pilot flying called “I visual”; 
however there was no similar call when the aircraft reached the MDA at 530 ft. 
At 500 ft, the EGPWS aural alert sounded “500” with no response from either 
crewmember. . . . By not calling out the standard MDA call at 530 ft, the crew missed 
an opportunity to ensure that the approach as still within all normal parameters at 
a point where they could execute a successful go-around (MAIB 2013). 


In this instance, even and especially when the acronyms are written out—minimum 
descent altitude (MDA) and electronic ground proximity warning system (EGPWS)— 
readers unfamiliar with the world will be lost even though every word is an integral part 
of the basic set of words commanded by an English speaker. This fragment is part of a 
text that has to stand on its own. Everything required for understanding the conclusions 
on the part of the target audience needs to be stated in the text. However, even though 
the text is closed in this sense it cannot be read without initiation into the world of 
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aviation. This world of aviation goes with and is presupposed by the text, even when 
it is not directly referenced: any part of the text makes reference to other parts. That is, 
without a sense of knowing your way around the world of aviation it is impossible to read 
and interpret a text (evaluation or accident report)—even though, as text, it inherently 
is to function among other texts in a world of texts. Because the world of actions can 
be theorized based on a model of the text, intelligible action may be theorized as text 
(Ricceur 1991). It is therefore worthwhile to extend the discussion of mimesis, to the 
relationship between understanding and explanation. Knowing about this relationship 
is important to understand cognition in the case of debriefing and assessment, because 
flight examiners provide explanations for the observations they make, such as in the 
preceding case where situation awareness, decision-making, and management are used 
to (a) explain a particular event in the simulator and (b) describe how pilots are to act 
such that they are extracted from the situation (i.e., management, setting priorities). 

The work on the cognitive processes underlying reading and interpretation suggests 
not merely a close relation of understanding and interpretation but indeed a mutually 
constitutive (also dialectic) one. Thus, practical understanding underlies explanation; 
in fact, it precedes, accompanies, and concludes any explanatory procedure. Practical 
understanding thus constitutes an envelope of explanation. This may appear self- 
evident, in that one cannot expect someone without a deep understanding of flying an 
aircraft to assess the performance of a pilot. Such understanding derives directly from 
the praxis of flying aircraft, giving any flight examiner a sense of what is to be done 
in a particular situation—independent of any theoretical knowledge they may apply in 
assessing the performance of others. This primacy of understanding over explanation 
is a first figure of the dialectic holding together the two processes of understanding 
and explanation. Here, explanation manifests itself in the form of a structural analysis, 
such as can be seen in the preceding example. The flight examiner talk does not just 
use some scale to rate the performance of the pilot but indeed suggests a relationship 
between management and situation awareness: situation awareness decreased with 
the consequences described because the pilot did not manage well, did not make the 
headsets (characterized as a distraction from the current task) a minor priority in this 
situation of a potential stall. Here, a cause (lack of management) was related to an 
effect (decreased situation awareness). In part, the reasoning also works the other way 
around: a lower level of situation awareness affected decision-making; and appropriate 
management could have improved both. 

In this model, a second movement takes us from the methodical moment of 
explanation to the non-methodical moment of practical understanding (i.e., knowing 
one’s way around the world). This movement constitutes a second figure of the 
dialectic. We can see from the preceding example that the explanation does something: 
it enhances understanding of the situation by providing—whether the flight examiner 
expresses this in a scientifically justifiable way is not at issue here—an explanation of 
actions, situation, and problems. That is, through the analysis of the situation in terms 
of the theoretical discourse, the flight examiner talk develops conceptual understanding 
exhibited in the descriptive account through the explanation it provides. Because 
understanding is displayed in the narrative account of the actions, it has nothing to do 
with the immediate grasping of the psychic life of the pilot or an emotional identification 
with the mental intentions of the pilot. Instead, the inherent intelligibility of this (as 
that of any reasonable) narrative account makes it objective. The practical, primary 
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understanding may be enhanced in the course of the analysis, which draws on an 
explicit human factors-based discourse. This new and more elaborate understanding 
expressed in the assessment then is premised on the procedural aspects of explanation 
that was preceded, accompanied, and concluded by the primary understanding (of how 
the world in the cockpit works). 


Application of Theory and Mimesis, 


When mimesis,, the order of the world as stated in text and theory, is brought back 
to the world of everyday practical action, which is the primary level of intelligibility, 
a third representational form is required. Whereas mimesis, is the representation of 
mimesis, in the world of text, mimesis, is the representation of mimesis, in the everyday 
world (Ricceur 1984). Mimesis, therefore constitutes the intersection of the world of 
text (theory, talk about praxis), analysis and theoretical explanation, and the world in 
which humans (in the present case pilots) act. Understanding and theorizing this form 
of mimesis is important because it allows investigators to grabble with the relation 
between debriefing meetings, where pilots come to talk about what they have done and 
learn from it, and their future praxis of flying aircraft, which is to be enhanced in and 
by the outcomes of debriefing. That is, improving aviation praxis is the raison d’étre 
of debriefing, its in-order-to motive, which requires a sound model for how debriefing 
talk translates into the everyday world of pilots. 

In the preceding subsection, it can be found that explanation enhances understanding. 
But an enhanced understanding itself does not yet mean enhanced dispositions for acting 
in the world. Understanding that the failure to prioritize stabilization of the aircraft 
through increased power and the resulting increase of speed does not yet assure that the 
pilot in question above will better manage such a situation the next time it comes up in 
the simulator or during a line flight. Here, we may draw on analogies with professional 
sports, such as golf. It is well known that when golfers intend to change their swing 
based on their and their coaches’ analyses, they first play worse before (if at all) they 
play better. That is, even among the most successful athletes, the step is difficult from (a) 
grasping what they have been doing and what they want to do in the future to (b) actually 
playing better. There are often months of work required to rework the dispositions (e.g., 
to swing) so that a better play results. In Chapter 8, differences are shown, explained, and 
discussed between the different effects artifacts have on learning through debriefing. To 
foreshadow the contents of that chapter let it be said that artifacts and talk more closely 
related to actual praxis provide more assistance to the implementation of new forms 
of practical understanding and the resulting praxis. Thus, rather than stopping with the 
low assessments of the situation awareness, decision-making, and management scales 
based on the airline’s conceptual model, the pilot in question would benefit from one 
or more action models. Thus, as discussed in Chapter 8, pilots may be taught to act 
according to a model captured in an acronym—e.g., the TDODAR (Time, Diagnosis, 
Options and associated risk, Decide upon action, Assign tasks, Review) that the British 
Airways pilots use or the F-DODAR model used in other airlines. Pilots then practice 
acting accordingly during the debriefing meeting what they are to do when facing an 
emergency situation in the simulator or at work. Over repeated instances of practicing, 
the entire action sequence eventually is produced without requiring consciousness. 
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e Whereas scientific and commonsense reasoning tend to take remembering as a 
more-or-less reliable outcome of brute mental skill, social psychologists have 
shown that remembering actually constitutes collective activity that is inherently 
(not merely incidentally) social. Lev Vygotsky, a founding father of social 
psychology, suggested over 80 years ago that humans control their brain from 
without by means of signs and tools that are inherently social. 

+ Debriefing meetings constitute a transactional ecology rather than the coming 
together and mutual influencing of independent actors and artifacts. 


190 Cognition, Assessment and Debriefing in Aviation 


e Human praxis and sensuous lived-experiencing are characterized by irreducible 
embeddedness in the world of activity. Because there is no time out for reflection, 
actors witness the world in particular, inherently incomplete ways, especially 
because they do not have access to a finalized view until some happening has ended 
and become a definitive event. At that point, aspects of what has been inchoate 
lived-experiencing may be made to jell into and grasped as an experience that 
stands out and becomes a memorable thing. 


e A lot of inherently cognitive work is required for constituting mind, memory, 
thinking, and reasoning in the course of a debriefing meeting. 


any pilots emerge from the four-hour simulator sessions for recurrent training 

or assessment purposes having forgotten most of the detail of what they have 

undergone. Even more-seasoned pilots, interviewed at the halfway mark 
of the simulator session or immediately following its end, say that little or nothing 
sticks out from what they have just emerged. If the purposes of recurrent training 
and examination also include learning from flying difficult situations to be able to act 
appropriately should these ever occur while doing regular line duty, then pilots need to 
remember what has happened first so that they can analyze it. If they do not remember, 
then they are not learning from what they have lived and undergone but, at best, from 
a text that might as well describe the performance of any other pilot. From the analysis 
of what will have become an experience, lessons can be extracted and learned so that 
pilots may draw upon them in their everyday work world. 

Experience is often treated as something private even though some philosophers 
and psychologists have pointed to the fact that there are non-private, objective as well 
as private, subjective dimensions to experience. Viewed psychologically, “learning 
from experience” tends to be accounted for as changes internal to individuals, going on 
in their minds. Anything analysts term to be social aspects of debriefing are treated as 
social in a weak sense, that is, as the results of more than one individual being involved. 
In this chapter, a view of learning from lived-experiencing is elaborated in which the 
phenomenon is social in a strong sense. A psychological phenomenon is social in a 
strong sense when its very genesis was a social relation first (Vygotsky 1989). Even 
when individuals manifest the psychological phenomenon while being on their own 
(as in writing into a private journal), it will be social (writing in the journal to oneself 
as to another). Precisely because the phenomena are social can they be unfolded again 
and accomplished in a social relation. This approach is taken here to the phenomenon 
of remembering and constituting what has happened in the form of an experience. 

From what has been an (inchoate) stream of lived-experiencing that comes with 
being caught up in the unfolding of life, aviation debriefing meetings isolate some part 
to make “an experience”, which can then serves as the material for analysis and the 
distillation of a take-home message that informs pilots’ future praxis. The term lived- 
experiencing here is used as “a name . . . to emphasize the interconnectedness of all 
concerns, affairs, pursuits, etc.” (Dewey and Bentley 1999), an interconnectedness that 
is characteristic of the ways in which human beings with their in-order-to motives find 
themselves caught up (acting) in the world. In aviation, any account of past, simulator- 
based events, that is, any account of “an experience”—containing first- and third-person 
parts—will emerge from the irreducibly joint work of all participants in the debriefing 
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meeting. This view on lived-experiencing then is consistent with the sense developed in 
Chapter 1: Because, as category, it names the unity/identity of person and environment, 
it is neither subjective nor objective. As an outcome of a cultural-historical, inherently 
societal activity, an experience also is always social in a strong sense. 

This chapter is concerned with the discursive work that participants in aviation 
debriefing perform to produce experiential and explanatory accounts on which are based 
assessment, training recommendation, and learning on the part of the pilots. A small 
collection of studies concerning the work of pilots focuses on the role of talk while 
actually piloting a multi-pilot aircraft. It includes studies of how approach briefings 
are conducted (Nevile 2004a), how pilots ensure timeliness and avoid overlap (Nevile 
2007a, 2007b), and how pilots explain diagnostic results (Hutchins and Palen 1997). 
Such studies explicate the different dimensions of the work of communication during the 
manipulation ofan aircraft. The concerns in the present chapter are different. The chapter 
investigates particular situations in the life of aviation when practitioners articulate the 
craft of flying in cognitive terms. Chapter 3 shows, among others, how flight examiners 
employ the discursive properties of terms such as situation awareness, maintenance 
of tolerances, decision-making, knowledge, management, and communication in 
the course of reasoning about pilot performance. In that case, fuzzy logic provides a 
model of the mapping that occurs between the observational (verbal) descriptions flight 
examiners produce while observing pilots in the cockpit and their scores for each of 
the categories (see Chapter 4). No assumptions are made here about what there really 
is in the mind of the pilots. 

A model is presented in Chapter 6 of the different ways of ordering the world that 
occur in ongoing praxis (mimesis,), in talking about, describing, and theorize praxis 
(mimesis,), and in translating formal or informal theories of practice back into ongoing 
praxis (mimesis, ). This is the first of four chapters that extends the pursuit of cognitive 
discourse in the aviation community but with a focus of studying the use of cognitive 
terms in the actual work of pilot assessment. When flight examiners say something like, 
“As soon as I understood the pilot’s thinking, it was obvious that it wasn’t a pass”, they 
co-articulate the presupposition that thinking can indeed be made visible in debriefing 
meetings. A second presupposition is that it takes the debriefing meeting to make it 
visible, and that in observing the performance in the flight simulator alone, the thinking 
was not exhibited or remained invisible. If thinking, cognition, reasoning, and other 
cognitive properties are made visible, then an ethnography of reason (Livingston 2008) 
may be employed to exhibit the ways in which “what a pilot was thinking” is made 
visible in and as the joint work of the participants in debriefing meetings. It is a study 
of cognition on the ground (Maynard 2006), where aspects of cognition are made to 
manifest themselves as social relations for the purpose of being recorded to serve as 
evidence on which further action is based. The cognitive events of interest here are 
understood to be “collaboratively organized, and bound up with distinctive instruments 
and objects” (Lynch 2006). The topics of pilots’ remembering, thinking, situation 
awareness, decision-making, and so forth are taken as matters subject to the active 
management of the talk in the debriefing meeting; and the focus here is “on examining 
in close empirical detail how common sense topics such as what people think, intend, 
feel or want, are handled and managed in everyday talk as part of everyday interactional 
business” (Edwards 2006). 
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DEBRIEFING AND COGNITION 
Remembering is Collective Work 


The study of memory has a long history not only in psychology but also, and before 
that, in philosophy (from Plato and Aristotle to more recent times) and in literature; 
and it also is an important feature of everyday life. In mundane, legal, scientific, and 
professional settings, memory tends to be treated as a storehouse (e.g., “long-term 
memory) from which some agent pulls representations of the past into working memory. 
This storehouse image of (long-term) memory is fraught with many logical problems 
and inconsistencies (Ricceur 2004). Thus, for example, a lot of what humans do, does 
not have to be represented in the mind because it is available in the world, where the 
human agents have access to it; the only requirement cognition must fulfill is that of a 
system of deictic codes that point the agent to the pertinent locations and phenomena 
(Ballard et al. 1997). Moreover, one special feature of humans is their ability to regulate 
the functioning of their brain by means of signs from without, which means, that rather 
than focusing on the structures of the brain we can focus on those collective practices 
that are used to control the brain, for example, allowing us to remember (Vygotsky 
1989). Sign use emerges in societal relations with others, and it is therefore “in society 
that people normally acquire their memories. It is also in society that they recall, 
recognize, and localize their memories” (Halbwachs 1992). Among those sign systems 
that allow humans to remember, and which in fact automatically fill in anything not 
directly remembered, are narratives. These are structured sign complexes that organize, 
and provide resources to, memory. The objective forms of culture, which include the 
forms of written and spoken language, constitute that which can be handed on (events 
of the past) rather than individual subjective memory or some kind of collective psyche. 

Memory constitutes a paradoxical phenomenon because it involves the 
contemporaneity of the past and the present (see Chapter 6). This contemporaneity is 
in fact a condition for consciousness of objects and events. Without the capacity to 
make the past present again, through some form of representation, humans would not 
be able to know that an object continues to exist even when it is absent. Thus, prior to 
some months into its life, a baby does not have the sense of object permanence. When 
an object disappears behind a screen, the baby will not look for it: the object no longer 
exists, as captured in the idiom “out of sight out of mind” equally applied to animals. 
The genesis of object permanence is tied to the emergence of the capacity to make 
something present in its absence; in infants it is tied to the relation with significant 
others who engage them in peek-a-boo and hide-and-seek games. 

Object permanence is the condition for any sense and conception of time, which 
is the phenomenon that brings about the change between what is and what has been 
(past), and, conversely, between what is and what will be (future). When events are 
recalled, there is therefore an interleaving of the recalled event and the event in which 
the recall happens. That is, the text making present the past or future (i.e., mimesis,) 
is intertwined with the here-and-now of the debriefing meeting that creates the text 
(mimesis, ). The interpenetration is produced by the fact that the language used in 
debriefing communication also recreates the preceding events: the same language makes 
the meeting (e.g., the social structure) and makes the text that the meeting is about (the 
reconstituted past, the object of reflective behavior). In the present chapter, the focus 
is on the co-presence and intertwining of the world of the narrators and the narrated 
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world in collective remembering (e.g., Middleton and Edwards 1990) during debriefing 
meetings. During these meetings, memory itself frequently becomes the topic, involving 
remembering, forgetting, and the quality and certainty of these processes. For example, 
after it has been established that a pilot had not attended to power (sitting at 0% torque) 
and speed (see Chapter 6), we may observe something like the following exchange: 


Scenario 7.1 


01 FE : You missed something in that situation. Do you remember what it was? 
02 PF : No. lam really tired now after the simulator session. My brainpower has 
gone. 


03 FE : I acknowledge that too much time has probably passed since then. Do 
you remember how you leveled when ALT capture occurred? Do you 
remember where your power levers were? 


In this fragment, there is an invitation to remember an aspect of an event in the 
flight simulator, and an indication that there was no memory of what this aspect might 
have been. There are also two explanations for the failure to remember: the absence of 
brainpower and the duration of the intermittent time. There is then an initiation to assist 
in recall by describing part of the situation (turn 03), and thus priming episodic memory 
(Roth 2017). As a result, whatever is produced as an account of the event—e.g., the 
fact that it was the PM who had increased power to attain the required speed—comes 
to be social in at least two senses. First, outside of the verbal exchange, the pilot flying 
might have never remembered what has happened during the talked-about event, or 
even that some such event had ever occurred. In this case, and despite having undergone 
a particular event, the pilot might have never learned anything from it. The account 
of the event as it has happened is the result of the communicative exchange, which, 
qua communicative exchange, is a social rather than individual phenomenon with its 
own internal dynamic (we cannot ever fix the outcome of a dialogue beforehand).! 
Most recollections produced on any given day are the consequences of questions 
others actually ask or could have asked; and the replies not only take up what others 
have queried but also explicitly address the perspectives and contexts of the intended 
recipients (Halbwachs 1992). Any recollection constituted for the purpose of a reply, 
therefore, irreducibly intertwines self and other: it is social through and through. 
Sociologists and social psychologists agree that there is no point of seeking for memories 
in specific locations of the brain, in the nooks and crannies of the mind, accessible to its 
owner, for they are stimulated and recalled externally (Halbwachs 1992; Vygotsky 1989). 

Second, the account is inherently social even if it had been entirely produced by 
the pilot, for example, in some “reflective journal” of the type that is used in (research 
on) teaching or in a variety of practitioner training programs (teaching, nursing). 
This is so because whatever is described and explained is subject to intelligibility. 
Intelligible accounts of events interconnect actors, situations, things, and relations into 
an unfolding story or plot. This story (plot) constitutes the text in the world of mimesis,. 
The production of an intelligible account, the auto/biographical emplotment of the pilots 
in their world that constitutes the narrative of what has happened, always is subject 
to cultural constraints: those that come from knowing one’s way around the world, 





' Dialogue used as a way to expound a pre-given truth, such as this is the case in the late Socratic 
dialogues, actually is monologic in its essence (Bakhtin 1984). 
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and those coming from the structure of language itself (e.g., the relation of subject, 
action-verb, and direct or indirect object in a standard phrase). The plot, which has a 
beginning, middle, and end, pulls the account together into a single whole. This makes 
it “that nothing is an event unless it contributes to the progress of a story” (Ricoeur 
1991); and the competency to follow the plot and story constitutes the complex forms 
of understanding characteristic of the culture. This cultural aspect is important, for it 
makes for the differences in the relations between pilots in the cockpit so that in many 
Asian cultures, first officers would be much less dispositioned to correct or question 
the actions and decisions of their captains. A case in point is found in the crash of 
Korean Air 801 in Guam, where the flight engineer, rather than telling the captain and 
first officer that the glideslope was not working (as per ATC communication) asked 
in a round-about way, “Is the glideslope working? Glideslope? Eh?” (NTSB 2000). 

The framework outlined here is important especially in the case of aviation 
debriefings, because, as the fieldwork has shown, pilots often describe what they have 
undergone as a blur. If pilots are to have any take-home messages then an important 
aspect of the debriefing meeting is the work of remembering what they have done and 
what has happened. This provides for the possibility to explain the events by bringing 
together the subjective sense of the pilots and the articulation of the objective task 
conditions, as described by the flight examiner. For those airlines that have access to a 
debriefing tool, additional means are available for the objective representation of critical 
events in the present of the meeting. The role of this important tool—despite the lack 
of evidence for the effect of video, which is part of the tool—is apparent in this and the 
following chapters that present specific cases of analyzing and teaching for improving 
the performances of pilots and cockpits (in the joint cognitive systems approach). 


An Ecological Perspective on Debriefing 


In Chapter 1, a model for understanding everyday practical action is presented. It is 
based on unit analysis that requires of any minimal unit of analysis to have all the 
characteristics of society. That system now, in debriefing, becomes the object of the 
activity (Figure 7.1). Its motive is to provide a narrative account of the cockpit system 
as a whole generally, and of the behavior of two of its parts specifically (PF, PM) 
during particularly crucial aspects of the tasks during the four-hour simulator session. 
For simplicity, only one pilot is depicted in the model. It shows how tools, artifacts, 
and talk present during the debriefing meeting all are part of the shared environment 
of flight examiner and pilot (Figure 7.1, top). The object/motive of the developing talk 
is the production of a descriptive, explanatory, and evaluative account of the preceding 
flight simulator events, where the two pilots and the system formed a cognitive ecology 
(lower left, Figure 7.1). The talk in the debriefing room, while being for and about 
something, also produces the context, the social relation in the debriefing room that 
binds pilots and examiners. 

It is self-evident that all aspects of communication are shared or there would be 
no communicative exchange at all and no relation: there would be no meeting but only 
a co-location of people. When it becomes apparent that something is not in common 
(e.g., understanding, a matter of fact), then it becomes the topic of talk. The language- 
in-use is closely tied to the world of the pilots and flight examiners. In the philosophy 
of language, unity/identity of activity and associated language is denoted as language- 
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Figure 7.1 In this ecological approach to cognition in debriefing, flight examiner and pilot each are 
part of each other’s environment, as are all the artifacts, tools, their respective talk. The object of the 
debriefing is to produce a narrative account, explanation, and assessment of the preceding simulator 
session, itself theorized in terms of an ecological model. 


game (Wittgenstein 1953/1997). That is, not only the language has to be common but 
also the activity where it is used and the world that it refers to. This is why pilot talk 
makes very little sense or no sense at all to a newcomer who is listening for a first 
time to a debriefing meeting. Not only the language itself but also the narratives are 
subject to constraints. To be intelligible and plausible, any account of a flight event 
has to include certain actors (pilots and aircraft system) and conditions (weather, the 
state of the aircraft, engines, and rudders) that relate in specific and highly constrained 
ways. Certain plots and stories about past events are plausible, whereas others would 
not be. Causes and effects are then attributable to the different parts of the unit as a 
whole, thereby contrasting lived-experiencing where effects are not available yet and 
therefore causes are undetermined. Even though it is impossible to assign causes and 
effects within a happening (i.e., prior to its conclusion when it becomes a definite 
event), as soon as it is completed and has become a denotable event, causes may be 
inferred from the effects (e.g., Dewey and Bentley 1999; Schiitz 1932). For example, 
in the case of the crash of a Fokker F100 near the Heho (Myanmar) airport, the pilot 
monitoring (captain) stated during his testimony that he had monitored the instruments 
in “head-down” condition (see box “Non-Directional Beacon Approach” below). This 
makes it intelligible and plausible that he did not see the aircraft entering instrument 
meteorological conditions (IMC); it is intelligible and plausible that he was not and 
could not have been aware of the change in the visibility. Whereas this does not abrogate 
his responsibility, and whereas it does not tell us about the truth of the matters, it does 
make his lack of awareness intelligible and plausible whatever had been the forever- 
irrecoverable situation at the time. 

The debriefing meeting provides a special opportunity to produce accounts that 
bring together two perspectives—the subjective perspective of the pilot acting in the 
heat of the moment and the objective, third-person perspective for which the flight 
examiner is a (fallible) representative. Flight examiners have set the conditions of the 
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flight—which are the same for all pilots in a given half-year training and examination 
cycle—such as varying weather conditions, types of failures, and so on. The givenness 
of these conditions provides constraints on the story line that pilots can provide for 
their actions. Thus, because the IMC condition was taken to be a fact, the PM of the 
Fokker F100 (see box below) had to address in his account his lack of awareness. On 
the positive side, articulations of external views constitute opportunities for pilots to 
grasp what has inexplicably happened. For example, a pilot might have witnessed that 
the aircraft did not “tick over” (make a turn) when it had reached a specific altitude at 
which the aircraft should turn according to the aviation chart. There might have been 
a second condition, as it exists for certain departures: the aircraft has to have crossed 
the end of the runway. That is, the global navigation satellite system (GNSS) would 
only direct the aircraft to the next waypoint if both conditions were met. Whereas there 
had been the awareness that the turn did not occur, the pilot becomes aware of the 
reason for the particular unfolding of the event only as part of the construction of the 
narrative. Simultaneously, the flight examiner becomes aware of the pilot’s perception 
and thinking at the time. That is, a more complete account of that departure includes 
the articulation of what the pilot was aware of and the objective conditions that the 
flight examiner knows to be the case from a spectator position. 

The unfolding narrative produced during and constituting the debriefing itself is 
a resource because, in its episodic characteristic, it assists the inherently social and 
collective re/construction of memory. Thus, flight examiners introduce a particular 
segment of the simulator session, such as naming a particular departure. They may 
begin producing a narrative, ask the pilot(s) to talk through the particular event, or 
direct the pilot(s) toward a particular artifact, such as the quick reference handbook, 
the representation of an assessment model, or a chart. Thus, flight examiners in the 
preceding departure scenario might have asked the pilots to read the departure chart. 
If the chart reads, among others 


e Maintain RWY CL[earance] to 500 ft 
+ LEFT turn at MNM 500 ft after crossing the RWY end 
e Trackto... 


then an opportunity is provided for remembering aspects of the departure in more detail. 
Pilots are facilitated to recognize the fact that there is a second condition, which is of 
special importance if the pilots were unaware of it up to this instant in the debriefing 
meeting. Not only will there be a narrative account of the events (implicit or explicit) 
but also there will be an explanation, which allows the pilot to grasp why the left turn 
did not occur, a grasp that is implicit in the statement, “So that is why the GNSS did 
not tick over”. 

The ecological perspective on the debriefing meeting recognizes that the talk has 
multiple simultaneous functions: it (a) produces the meeting, (b) makes apparent the 
order of the meeting and how this order is produced, and (c) produces a narrative account 
(representation) of past events that have unfolded in the simulator. Any transcription of 
a debriefing meeting will exhibit the interleaving of all three tasks. Take the following 
fragment, where the flight examiner introduces the event that had occurred by naming 
an important artifact involved: the engine-fail checklist. 
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Scenario 7.2 


01 FE : Youactually missed something important in the engine-fail checklist. Do 
you remember what it was? 

02 ((Long pause)) 

03 FE : You missed the auxiliary power unit. You never started it. 


04 PM: Can we have a look? ((Pointing to quick reference handbook, ORH)) 

05 FE : Have we got the quick reference handbook? ((Gets ORH)) 

06 PF : That checklist was broken up. You hit the button but never actually 
started it. 


In this scenario, the present and past are intertwined. First, some parts of the talk 
belong to what has happened in the past, the missing of something (i.e., the engine fail 
list, the auxiliary power unit), the naming of what has been missed, and the action that 
has never been accomplished. But, second, there is also an invitation in the present for 
doing something: an invitation to remember and, implicitly, to articulate that which has 
been remembered and invitations to get the QRH involved. This clearly is an aspect of 
the organization of the debriefing meeting itself. Rather than beginning an account or 
attempting to remember or stating that the event had been forgotten, the PM requests 
having a look, which apparently is a look for something that is to be found in the quick 
reference handbook. The reply also concerns the organization of the debriefing meeting, 
an orientation to the QRH. The next turn quite apparently continues the narrative of 
the event, articulating that there were interruptions while the checklist was read and 
actions carried out, and, whereas the auxiliary power unit (APU) button was hit, it 
never actually started the APU (turn 06). Remembering, having a look, and locating 
the QRH all are part of the meeting work, and the talk that occurs exhibits what is to 
happen, what is going on, and often what has happened in the meeting before (e.g., “we 
have talked about this before”). Something like the invitation to remember both make 
an aspect of the meeting happen, produce the meeting, and exhibit the order of what is 
happening: whatever follows is remembering work. Nested within this remembering 
work is the organization ofthe access to a resource, the QRH, the tool that was actually 
used during that original situation in the simulator; that tool was present in the cockpit 
and is present again in the debriefing meeting. The original situation, which at the time 
did not have closure, now is something that can be named, for example, “When you 
missed starting the APU”. This naming invites cutting something out from what was 
unfolding as a stream of lived-experiencing to achieve a more-or-less clearly defined 
and therefore definite (micro-) event. 


FROM SENSUOUS (INCHOATE) LIVED-EXPERIENCING TO AN 
EXPERIENCE 


In the simulator, pilots are caught up in and are integral part of what is happening, 
when the event as something completed and namable does not yet exist. In that mode 
of being, one duration flows into the next, and pilots are subject and subjected to the 
conditions that they contribute to producing. Because the in-order-to motive of flying 
does not include the production of a historical (auto/biographical) narrative, pilots do 
not generally make present again any objects actually present. They may act toward 
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these objects without requiring them to appear in conscious awareness—as shown in 
the preceding discussions of manually shifting gears or using a keyboard. As a result, 
they forget much of what has happened during their recurrent training and examination 
sessions. The work of debriefing then can be thought of as transforming what has been 
a sensuous and inchoate lived-experiencing—a single, whole being-caught-up-in- 
undergoing-of-and-acting-in-situations-out-of-complete-control—into an experience, 
something that stands out not only as a whole (e.g., “we crashed”, “we buggered up the 
NDB”) but also in its specific details that led to the ultimate outcome (crash, buggered-up 
NDB). There is a holistic quality to the sensuous lived-experiencing that only partially 
transfers into the finished experience as reconstituted in verbal accounts, which largely 
are intellectual (Dewey 1934/2008). Even if situation details subsequently are forgotten, 
an overall sense of a “buggered up NDB” or a couple of incorrect turns onto a 10-mile 
arc and into a hold (Chapter 5) may remain for months and years to come. 

It is acommonly accepted aspect of life that in everyday practice, humans change 
and learn, becoming better at what we do just by doing it: sensuous lived-experiencing 
changes subsequent sensuous lived-experiencing. Whereas this aspect is not captured 
in digital computers, and therefore also is absent from the associated way of thinking 
about human cognition (learning has to be assessable), it is inherent in all artificial 
neural network approaches, where a network undergoes change each time activation 
energy moves through it. Sensuous, lived-experiencing is inchoate, a continuous 
unstructured stream of witnessing without a finalizing grasp of what is going on. But 
sometimes, a particular stretch from that stream, once completed, transcends the stream 
and becomes “an experience”, which stands out in human consciousness as something 
“with its own individualizing quality and self-sufficiency” (Dewey 2008). In many 
professions, including aviation, practitioners engage in reflective practice as a way of 
learning from experience. This leads to a reconstruction of (sense) experience so that 
it can become “an experience”, which is a precondition for experiential knowledge: an 
experience as a condensation of knowledge. Indeed, there cannot be intellectual growth 
without work, without some reconstruction, some remaking, of what was apparent in 
the stream of life as recalled after it has been completed. Such reconstruction requires 
going beyond the continuous stream of perception and lived-experiencing, which 
can really be transformed only when lived-experiencing is considered in terms of the 
cultural-historical conditions that shape it. This is so because an experience, by its very 
definition (see Chapter 1 and preceding section), involves the interplay of objective 
and internal-subjective conditions. Unless these objective, historical conditions are 
known, participants are caught within the horizons of their own worlds and may never 
grasp what is happening to them as such (Holzkamp 1983; Smith 1999). Individual 
development requires transcending concrete experiences and coming to understand the 
external determinations of the conditions that shaped the lived-experiencing. 

To be able to learn and reflect upon a preceding stream of inchoate lived- 
experiencing, so that it becomes some definite (an) experience that stands out from 
everything else in the stream of life, the event as it has been witnessed needs to be 
present again. Such re-presentation is the condition for the identification of anything 
humans witness both in the other as in the self: no pure presentation is ever possible 
because self-presentation and re-presentation inevitably are interlaced (Franck 1981). 
Development then occurs when previous lived-experiencing (E, in Figure 6.2) becomes 
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the object of consciousness of an ongoing, present lived-experiencing (E, in Figure 
6.2). But lived-experiencing is inchoate. To arrive at an object requires thingifying 
some aspect of lived-experiencing, requires cutting something out from what has been 
a flow. This means that whatever has happened has to have finalized such that it indeed 
is part of the object of reflection. 

Participants generally and flight examiners specifically understand the debriefing 
meeting as an opportunity for learning. Flight examiners, following a generic testing 
protocol, have subjected the pilots under examinations (generally a captain and a first 
officer) to a variety of unexpected situations—weather conditions, malfunctions of 
aircraft (e.g., engine failure, aileron jam), or malfunctions of instruments—and have 
observed what the pilots did and how they extracted themselves from the potentially 
calamitous situations faced. Flight examiners have taken notes and, where available, 
entered codes into a debriefing tool, which allows rapid identification of relevant 
segments from the examination together with showing in detail some aspects of the 
cockpit. The pilots, on the other hand, have been subject to the situations, which are 
differentially reflected in their consciousness, especially because they are positioned 
differently to the events that have occurred. To create the conditions for transcending the 
immediacy of previous inchoate lived-experiencing, to move from lived-experiencing 
(Erleben) to knowledge gained from an experience (Erfahrung), the pilots need to 
become aware of the interplay of objective and internal conditions of the event/s under 
discussion. This chapter thus is also concerned with providing answers to the question: 
What is the joint work of making an experience? What is the work by means of which 
the pilots’ first-person accounts of their lived experiences are coordinated with the flight 
examiners’ third-person accounts? How are the technologies (whiteboard, debriefing 
tool) used involved in this work? 

The term /ived-experiencing is taken to contain all the characteristics of the 
practically engaged person and all of the characteristics of the environment, which 
comprises practical, intellectual, and affective colorings. Although the category includes 
the external environment, the same objective (societal, material) conditions are refracted 
differently in the consciousness (intellectual) and affect of the individuals. Some artificial 
intelligence scholars refer to this world as lifeworld (Agre and Horswill 1997). Each 
agent therefore has some form of implicit horizon within which the objective world 
appears as the situationally relevant lifeworld. For example, the flight instruments of Air 
Bagan W9011 during the NDB-approach to Heho (Myanmar) objectively showed that 
the aircraft was not on the correct course (see box); but in the pilot perception, they were 
on course. Similarly, the engine instruments in the TransAsia GE 235 crash objectively 
showed which engine was damaged and which one was live (see box, Chapter 10); and 
yet, the PF shut down the live engine. Thus, these pilots acted within a perceptual horizon 
that differed from the objective world subsequently reconstructed in a scientific manner. 
Although the perceptual horizon differs from the scientific description, seasoned (social 
science) analysts can reconstruct the horizons even from very incomplete data such as 
talk alone and with very high fidelity (e.g., Roth 2015d). Flight examiners, constrained 
by the one-hour limit of the debriefing meeting, know about what the pilots have been 
aware of only through a questioning intentioned to uncover the conditions that shaped 
the pilots’ awareness of the perceived thing. 
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Non-Directional Beacon (NDB) Approach 


The Federal Aviation Administration (FAA) of the U.S. Department of Transportation 
intends to reduce the aviators’ dependence on non-precision approaches, such 
as the VHF omnidirectional range (VOR), non-directional beacon (NDB), and 
other ground-based NAVAID approaches (FAA 2015). Yet in 2015, its Instrument 
Procedures Handbook still notes the existence of 1,105 VOR and 916 NDB stations. 
In the US, the NDB approach is to be phased out gradually until it no longer exists. In 
other parts of the world, however, where there still may be many remote airports, the 
NDB approach may continue to be the only option. When the NDB does not involve a 
final approach fix, then the FAA requires including a procedure turn. In some airports, 
a teardrop-shaped turn characterizes the NDB approach. NDB approaches have been 
part of a number of relatively recent crashes, including that of a Fokker F100 on the 
flight into Heho (Myanmar) airport and a military Boeing 737 aircraft approaching 
Dubrovnik (Croatia). Although the details of the two accidents differ, both cases 
involve aircraft failing to stay on the published approaches, a fact that should have 
been apparent from the bearing and automated direction finder instrument. 

In this chapter, the crash landing on the NDB approach to Heho (Myanmar) 
is used as the basis for a simulator scenario closely matching the accident. What 
a subsequent debriefing might focus on to work out the human factors involved 
constitutes the content of later part of Chapter 7. Information concerning the crash 
was gathered from the final report of the Myanmar Accident Investigation Bureau 
(MAIB 2013), a report in The Aviation Herald (Hradecky 2013), and an article in 
the Myanmar Times (Than 2013). 


Air Bagan Flight W9011 


On Christmas Day in 2012 at 8:50 local time, Air Bagan Flight W9011 from 
Mandalay to Heho, at that time using a Fokker F100, struck a west-to-east-running 
66 kV power line about 1 nm south of the Heho (Myanmar) airport. As the map 
(Figure 7.2) shows, the aircraft came to rest about 750 m after the strike and after 
having taken down the tops of trees along National Highway 4 (which severed the 
wings), also killing a motorcyclist passing by at the moment. After coming to a stop 
in a dried up rice paddy, the aircraft broke up and, seconds later, burst into flames. 
Among the 71 people on board (63 passengers, 47 of which were foreigners, 6 
crew, and 2 security personnel), there was one casualty and 11 injured. One of those 
severely injured later would write a book about the trauma from the burns he had 
received, about the mayhem following the escape from the burning plane, and his 
process of recovery from the heavy burns received while getting stuck during the 
escape (Lokos 2015). Many passengers initially were unaware of the fact that the 
aircraft had crashed, having felt only a minor bump followed by a few heavier ones. 
The tail section had fractured and separated; and it was here that the fire had started. 

The photos shot at the site shortly after the crash (about two weeks) show tree 
trunks and an 11kV power line next to Highway 4; the tops had been severed from 
the trees and the power line posts. The individual who had taken the shots did not 
find any traces in the field that the aircraft crossed prior to the Highway 4. All of this 
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Figure 7.2 This map of the accident site locates National Highway 4, where the killed motorcyclist 
was riding, the 11 and 66 kW power lines that the aircraft clipped, the runway, and the more densely 
inhabited areas. 


evidence suggests that the aircraft had made contact with ground on the highway 
itself, where a passing motorcyclist was killed following the impact with the aircraft. 

General weather information was not available for Heho. The nearest station, 
about 13 nm to the east of Heho airport, reported 12°C, dew point at 10°C, which 
corresponds to a relative humidity of 83%. There was light mist, winds from the 
southeast, and visibility of 6 km. The aircraft had contacted the Heho air traffic 
control, which provided the extant weather conditions: “wind calm, visibility 
3000 m, distinct fog, temperature 17 C, QNH 1018 mb, R[un]W[ay] 36” (MAIB 
2013). According to eyewitnesses on the ground, there had been heavy fog at the time, 
entailing low visibility, requiring vehicles to use headlights. Available photographs, 
shot at the scene immediately after the accident, do not show heavy fog, especially in 
the direction away from the airport. Whereas the airline first suggested that there had 
been an emergency landing, authorities later suggested that the pilot had mistaken 
(due to the poor visibility caused by bad weather at the time) the orthogonally 
running Highway 4 for the runway. 

The airport offers runways 18 and 36, with NDB approaches published by 
the Aeronautical Information Services Myanmar. The aircraft was to have taken a 
tear-drop-shaped approach to runway 36 (Figure 7.3). In this approach, the aircraft 
crosses the beacon and heads outbound on 220° and returns inbound on the 010° 
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Figure 7.3 Reconstruction of the published NDB approach plate information for the Heho 
(Myanmar) airport runway 36 (based on MAIB 2013). (a) Teardrop-shaped turn. (b) Relatively 
steep descent to minimum descent altitude. 


radial (Figure 7.3a). At its furthest point, the aircraft was to be 5.7 nm from the 
runway threshold. The missed approach point on the steep 5.2% descent (MAPt) is 
at 1.5 nm from the airport at an aboveground altitude of 540 ft (Figure 7.3b). The 
Myanmar Times quotes the deputy director general of the Myanmar Department 
of Civil Aviation, who attributes the causes of the crash to the pilot: “The accident 
occurred due to the incorrect judgment of the pilot. . . . When we checked the 
flight data recorder, it showed that the rest of the journey was fine and the accident 
happened only due to his misjudgment while landing” (Than 2013). The following 
excerpts from the final report can be found online. 


The captain made the briefing and completed the aircraft checks. At 08:26 
local time, the flight departed Mandalay to Heho Airport. The first officer was 
designated as the Pilot Flying for the flight. The aircraft climbed to FL130 and 
the captain contacted Heho ATC at 50 nm from Heho. Heho ATC provided 
the present weather condition (wind calm, visibility 3000 m, distinct fog, 
temperature 17°C, QNH 1018 mb, runway 36 in use). At about 08:36 local 
time, the first officer started the crew briefing. The aircraft started descend to 
9000 ft and continued to overhead the Heho NDB. At about 08:47 local time, 
while heading 220 degrees and descending to 6000 ft the flight commenced a 
non-precision Non Directional Beacon (NDB) approach to runway 36. 


During the final inbound track at about 2.5 nm to the runway at 08:52, the 
EGPWS [enhanced ground proximity warning system] aural warning called 
out “500”. The captain initiated “Alt hold” at about 08:53, just before the 
EGPWS alert “100” “50” 40” “30” and the aircraft struck 66 kV power lines, 
trees, telephone cables, a fence and collided with terrain short of the runway, 
coming to rest approximately 0.7 nm from the threshold (Aviation Safety 
Network 2016). 
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The actual flight path of the aircraft, as reconstructed from various pieces of 
information available in the final report—e.g., the actual flight path over a Google 
map of the area reconstructed from the longitude and latitude information, the 
nominal approach, the flight data recorder information on altitude, and the cockpit 
voice recorder information—deviates considerably from the published teardrop 
approach (Figure 7.4a). The area map shows that the aircraft turned too early onto 
a 220° radial and remained too far to the west of the nominal outbound course. The 
aircraft turned onto 110° for about 2 nm before turning onto 050° until intercepting the 
nominal 010° approach at about 2.3 nm from the runway threshold; this is well after 
the airline’s 1000 ft decision point for the IMC conditions that ATC communicated 
to the aircraft, but ahead of the 500 ft required for visual meteorological conditions. 
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Figure 7.4 This to-scale comparison between published approach and actual approach combines 
information from the actual flight path, the flight data recorder, and approach plate. (a) Actual 
horizontal track as per Figure 11 in the final report (MAIB 2013). A +5° field has been added 
to show where the aircraft was to be stabilized for descending below minimum approach point. 
(b) The actual vertical flight path according to flight data recorder mapped against published 
nominal glide path. 





The vertical flight path according to the flight data recorder, too, exhibits 
considerable deviations from the published approach (Figure 7.4b). The aircraft 
passed the beacon 1000 ft below the nominal 7000 ft, and descended 150 ft below 
the nominal 5490 ft at the apex. Starting at 4.5 nm from the runway threshold to 
the site of the crash, the aircraft descended the 1500 ft at a greater rate than the one 
published in the approach plate. 

Both pilots provided testimony. As to the final part of the approach, the pilot in 
command (captain) functioning as the pilot monitoring (PM) provided the following 
account: 


At the approach site of RW 36, he saw some cloud and decided Non-precision 
approach. Overhead NDB let down and called again field insight. At 7000 ft, 
intercept outbound 220° track and descended to 6000 ft and landing gear down, 
lift dumper arm and set flap 25° and intercept inbound track 010 degrees. As 








204 Cognition, Assessment and Debriefing in Aviation 


for airfield insight visually, PF instructed to select landing flap 42°. There was 
some cloud between runway and aircraft, while PF monitored the instruments 
in head-down condition aircraft passing in the cloud. The Pilot in Command 
instructed PF for “Alt hold” as the same time call out EGPWS “100”. The Pilot 
in Command pushed altitude knob to get level flying at Minimum Descent 
Altitude MDA and intended to disconnect autopilot and manual go around. 
But he felt abnormal sounds and impact (MAIB 2013). 


For this flight, the first officer was pilot flying (PF). He provided the following 
information: 


At 45 nm to Heho .. . MDA set to 4380 ft . . . plan to NDB approach. ATC 
approval to 7000 ft and overhead outbound 220° track and descend to 6000 ft. 
Then turn inbound track 010° and initiate descend to MDA with initial vertical 
speed 1000 ft per min. At 6 nm, runway was in sight and radio altimeter was 
showing 1800 ft. At 5 nm aligned with runway and flap down 42° and approach 
speed 139 knots. At about 4 nm, reduce vertical speed to 700 ft per min and 
passing the cloud. At 3 nm pressing the alt hold button and call out EGPWS 
100 ft. Suddenly he heard noise and impact (MAIB 2013, language corrected). 


When the two testimonies are compared to the actual flight path reconstructed 
from the flight data recorder, deviations are apparent. Most importantly, the aircraft 
was not aligned with the runway at 5 nm but only at about 2.3 nm. Similarly, flap 42 
was actuated at about 3.2 nm rather than at 5 nm from threshold. The PM reported 
that the “Alt hold” call was made just prior to the EGPWS call “100”, that is, less 
than 1 nm from the threshold and immediately prior to making contact, whereas the 
PF reported pressing the “Alt hold” button at 3 nm. 

According to the final report, the primary cause included (a) the aircraft’s 
descent below minimum descent altitude (MDA), (b) the non-compliance with 
company standard operating procedures, and (c) the failure to change from visual 
meteorological conditions (VMC) to instrument meteorological conditions (IMC) due 
to fog. The secondary causes for the accident include (a) the fact that the captain of 
the aircraft had insufficiently assessed the risk of designating the first officer as pilot 
flying and (b) the possibility of pressure on the crew given that Air Bagan 9011 was 
the first plane on that morning to be followed immediately by several other aircraft. 

The three safety recommendations included two pertaining to Air Bagan, which 
are also relevant in the context of the present book. Accordingly, the operator is to 
ensure (a) “the qualitative requirements of operational personal with non-precision 
NDB approach training with IMC, awareness of MDA, and EGP WS alert” and (b) 
“to access multi-crew operation of CRM training, supervision of captain and the 
risk FO to perform the PF” (Aviation Safety Network 2016). 


Heho NDB Scenario 


The following scenario follows the Heho NDB approach and uses the information 
that led to the crash. It serves as the background to the analyses provided in 
Chapter 7, which is concerned with understanding cognition during debriefing 
generally and the construction of an account that combines first-person and third- 
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person perspectives required for understanding pilot cognition as presented in 
debriefing meetings and how pilots learn from experience. For the present purposes 
it is assumed that a crew, during simulator training, has flown the approach in the 
same way as the crew in the Air Bagan W9011 accident. The debriefing case would 
thus concern airlines, such as Air Bagan, who are tasked with or decide on their own, 
to implement “operational personal with non-precision NDB approach training with 
IMC”, as specified in the recommendations of the Air Bagan accident report. The 
following information is provided so that readers unfamiliar with the NDB approach 
more easily make sense of the debriefing case featured and analyzed in Chapter 7. 

In the NDB approach, the automatic direction finder (ADF) is the key 
instrument, as it always points to the beacon, the location of which is indicated by 
the concentric broken circles. In situations where the beacon is situated on the airport, 
as in the Heho case (Figure 7.3a), the ADF therefore always points to the airport. 
The second instrument is the directional gyroscope that tells the pilots the direction 
in which the aircraft is headed. In older aircraft, the two instruments are found in 
different locations on the instrument panel. In newer aircraft, the two instruments 
are collocated, integrated into an electronic horizontal situation indicator; and they 
are always collocated in glass cockpits. The Fokker F100 involved in the accident 
has an electronic indicator the basic elements of concern in this scenario are shown 
in Figure 7.5. When the two instruments are not collocated, additional effort is 
required because, as instructors used to recommend, the pilot had to mentally 
locate the pointer of the ADF on the directional gyroscope. In difficult situations, 
characterized by crewmembers as providing a “high workload”, the additional mental 
resources required for mentally collocating the instruments decrease attentional 
resources required for other parts of the flight. In the scenario, the two instruments 
are collocated. However, for the present purposes a more generic version of the 
instrument is used in the scenario. 


automatic direction finder 
„= heading bug 





course ! >... directional 
deviator bar gyroscope 





Figure 7.5 Key elements of the Fokker F100 instrument that collocates the directional gyroscope, 
the automated direction finder, and a pointer for the desired flight path. 
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In the Heho scenario, when the aircraft flies on the nominal 220° outbound 
course of the teardrop, the ADF pointer would be oriented exactly opposite 
to the aircraft (Figure 7.6a). If the aircraft flies on the 220° (or a bit larger) 
radial, then the ADF would not be aligned with the aircraft. In the case of Air 
Bagan W9011, with the aircraft at position $3 on the outbound (Figure 4a), it 
would be about 10 degrees off (Figure 7.6c). Moreover, whereas the alignment 
would be constant throughout the 220° course on the published course, the 
ADF changes direction because the angle between the aircraft and beacon 
continuously changes. Conversely, on the 010° inbound course (position $5, 
Figure 7.4a), the aircraft, the desired track, and the ADF pointer are exactly aligned 
(Figure 7.6e). Thus, during the circling maneuver, when the aircraft flies at the 
apex (position #2, Figure 7.4a), the aircraft is headed towards 115 degrees, the 
ADF pointer is at 90 degrees to the left, leading to a difference of 15 degrees 
with respect to the 010° inbound course, here marked by the (orange) triangle 
(Figure 7.6b). Readers easily deduce that in the scenario, with the aircraft approaching 
approximately crossing the 205° radial (position 74, Figure 7.4), the configuration 
of the gyroscope and ADF would look like Figure 7.6d. Differences are apparent 
in the gyroscope and ADF positions (a) between the correct 010° inbound (Figure 
7.6e) and the incorrect inbound course in the scenario (Figure 7.6d) and (b) between 
the correct position crossing the 205° radial (Figure 7.6b) and the actual position 
when the 205° radial is crossed (Figure 7.6d). 
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Figure 7.6 Readings of the directional gyroscope and the automatic direction finder (needle) in 
five situations of the Heho teardrop NDB approach (positions as indicated in Figure 7.3a). (a) On 
the nominal 220° outbound track. (b) On the 020° radial at the apex of the teardrop. (c) On the 
actual outbound. (d) At the point where the aircraft was to be stabilized under IMC conditions (at 
1000 ft). (e) On the nominal inbound course and stabilized aircraft at 1000 ft above ground in IMC 
conditions. The (orange) triangle indicates the radial of the nominal 010° inbound. 
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In this scenario, because the aircraft is not established within + 5 degrees of the 
published approach at 1000 ft above ground under the IMC conditions at the airport 
(as per ATC communication), an automatic missed approach is called for. If such 
a case occurred in the flight simulator during training or examination conditions, 
the flight examiner would then acknowledge that for whatever reason already 
apparent or to be established during the debriefing—e.g., the failure to account for 
wind, or the simple failure to enact the second 90 degree turn too soon—the error 
was captured and the right decision was made to call missed approach and to fly a 
go-around procedure. Thus, in the scenario, the flight examiner would have to be 
concerned with the fact that the crew continued the approach. In the scenario, the 
flight examiner knows the actual course of the aircraft—in the case of an airline 
with a debriefing tool, could actually display the actual course on a geographical 
map, the actual versus nominal descent profile (as shown in Figure 7.4b), and the 
tracking of the aircraft with respect to the nominal approach (as shown in Figure 
7.4a). The purpose of the debriefing is to find out about the thought processes of 
the crew for the purpose of identifying the human factors involved and having 
been at the basis of the problematic flight. Retraining would be designed such as 
to remediate through additional training those human factors that were identified 
as having caused the performance that eventually was judged to have been below 
minimal performance levels. 











THE WORK OF RECONSTRUCTING REASONING DURING A FAILED 
NDB APPROACH 


Debriefing is intended to analyze and explain specific events that have occurred in the 
recent past—1.e., that have occurred in the simulator session preceding it. Language, 
however, does more than simply refer. That is, in addition to referring to (being 
about) something, the text of any utterance is part of five additional sense-constituting 
contextures related to language as such and to the pertinence of speaking like this, 
in this situation, to this intended recipient (see Chapter 6). Therefore, memory and 
representations of the past come to be interwoven with the work of producing the 
narratives that provide the basis for the assessment of pilots at the end of the meeting. In 
the overlap between the flight examiners’ observations in the simulator and the debriefing 
is inherent in the task of assessing past pilot performances based on specific human 
factors, which, depending on the airline involved, may include situation awareness, 
decision-making, technical aspects of flying the aircraft, knowledge and procedures, 
communication, management, airmanship, cooperation, leadership, and others. In the 
simulator, the flight examiners only have available overt behavior and the products 
of those human factors presumed, so that particularly in the case of the non-technical 
aspects, inaccessible directly, they might wait until after the debriefing meeting before 
assessing those skills. In certain critical cases, flight examiners might be unable or 
unprepared to make an assessment about pass or failure immediately after the simulator 
session. Their declared intent would then be to find out about pilots’ thinking, articulated 
as in-order-to motive prior to and during the meeting; the work conducted together 
with the pilots is concerned with making aspects of the mind visible so that they can 
be assessed in a reasoned manner. Whereas it is impossible to ascertain whether this 


208 Cognition, Assessment and Debriefing in Aviation 


articulation of thinking actually was the thinking during the flight event, whatever 
will have been articulated in the debriefing meeting constitutes sufficient grounds for 
the basis of the assessment and the reports that ensue (e.g., for the pilots’ file, for the 
training managers). The production of the accounts generally and the reconstructed 
reasoning in particular—understood in terms of the airlines’ explicit or implicit human 
factors models—takes work. In the following, aspects of this work are presented and 
analyzed in the case of a debriefing scenario with the pilots that have flown the Heho 
scenario as sketched (see box). 

From a flight examiner’s perspective, interested in figuring out how to assess the 
presumed skills and knowledge, the main purpose of the meeting is to understand the 
pilots’ thinking. That is, flight examiners already know what has happened from their 
observations and what the pilots have actually done. But, tasked with an assessment of 
the individual pilots’ skills, they may require knowing more than what can be seen from 
the pilots’ actions. The meetings may reveal that one or both pilots do not remember 
details of the flight or what they “remember” may actually not be consistent with the 
facts (the case of the Heho crash pilots, see box). The joint work of a meeting therefore 
is required for establishing a correspondence between whatever “thinking” is revealed 
and the observations made (or, if available, what is seen on a debriefing tool). However, 
saying that any outcome was “social”—e.g., the product of a “social construction” — 
would mean emphasizing the trivially social dimensions of the event. More importantly, 
the social character of thinking is neither incidental to the debriefing meeting nor a 
matter of the argument and empirical evidence that is mobilized here. Instead, the 
social character is present to the participants in the meeting as the witnessed thinking 
of pilots; and the social character of the thinking is in fact required to be witnessable 
by flight examiners and researchers alike. Thinking is social precisely because it can 
be exhibited in the public arena of a debriefing meeting. If it could not be exhibited, 
it would not be thinking of the verbal kind, that is, of the kind also accessed in expert 
think-aloud protocols. In other words, if it could not be exhibited then it would be 
unintelligible talking-in-tongues. 

Debriefing is work, which requires thinking (cognition). Focusing on work allows 
us to provide objective evidence of all the requirements for debriefing to be pulled off 
generally, and the requirements for identifying aspects of the pilots’ minds (knowledge, 
skills) specifically. In this section, six dimensions of this work are described: (a) 
remembering and articulating thinking in the “walk-me-through” mode, (b) articulating 
forgetting and lack of knowledge or awareness, (c) initiating remembering, (d) 
establishing the actual flight track and the associated reasoning, (e) teaching | learning, 
and (f) articulating the performance in terms of human factors. 

The basis for the description and analysis is the scenario of a debriefing meeting 
that might have occurred if a crew had flown the approach to Heho in the same manner 
as the crew in the case of Air Bagan Flight W9011 (see box). The features of the meeting 
described constitute a composite of the kinds of things flight examiners and pilots do 
and say during actual debriefing meetings about the kinds of issues represented in this 
scenario. The important aspect here is the description of the kind of work that flight 
examiners and the assessed pilots routinely perform during their biannual recurrent 
training and assessment sessions—especially when the question is what the pilots had 
been thinking privately so that it was not available as shared consciousness in the cockpit. 
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Remembering and Articulating Thinking in the “Walk-Me-Through” 
Mode 


Debriefing meetings often start with an opportunity for the pilots to state significant 
parts of the simulator session. Thus, invited to walk the flight examiner through the 
NDB-scenario LOFT and to state what was done, the pilot flying (first officer) in the 
Heho scenario says what the actual pilot stated in his testimony. The testimony here 
is given as one chunk of text, whereas in actual debriefing meetings flight examiners 
produce signs of acknowledgment, or ask the other pilot questions like, “What did you 
think of that decision?” 


I got ATC approval to descend to 7000 ft and continue overhead outbound on the 
220° track, descending to 6000 ft. I then turned onto the inbound track 010° and 
initiated descend to MDA with an initial vertical speed of 1000 ft per min. At 6 nm, 
I saw the runway and the radio altimeter showed 1800 ft. At 5 nm, I was aligned 
with the runway, had flaps down to 42°, and my approach speed was 139 knots. 
At about 4 nm, vertical speed was reduced to 700 ft per min and we were passing 
cloud. At 3 nm I pressed the alt hold button and we heard call out EGPWS 100 ft. 


After this account, flight examiners already know that there are issues: the 
aircraft was not aligned with the 010° inbound until 2 nm from the runway threshold 
(Figure 7.4); the descent was at 1,000 ft/min right to the end (Figure 7.4), and, whereas 
the air craft was aligned with a 220° outbound track, it was not aligned with the 220° 
specified on the approach chart as seen from the automated direction finder (Figure 
7.6c). In the facilitated briefing approach, flight examiners would provide the time, 
and pilots take their time, required for the production of an account of the day in the 
simulator. The turn to the pilots are explicitly formulated and offered as an opportunity 
to remember; and in ending, the pilots formulate what they were doing and what was 
going on in their minds. Already such invitations to walk a flight examiner through the 
day, and this walk through being formulated as what is being remembered from the 
simulator session, constitute the framing of the debriefing meeting in terms of cognitive 
processes. What a pilot was thinking at the time is related to what is remembered, and 
considerable additional remembering work is required to establish what the thought 
processes had been. Given the difference between the account that the PF provided and 
what the flight examiner knows to have happened (i.e., the actual horizontal and vertical 
positions of the aircraft as in Figure 7.4), a lot of additional work is required to reveal 
thinking, reasoning, or decision-making for the purpose of helping flight examiners 
make up their minds about whether to pass or fail the pilot (i.e., for assessment purposes) 
and what kind of recommendations to make for retraining purposes. 

Some readers might think that the Heho scenario presents a much-too-clear case 
of a failed examination to be uncertain. During the fieldwork, however, repeated 
instances have been observed where pilots ultimately passed even though they had 
taken a wrong turn or where pilots were having precisely the kinds of issues as the 
pilots in the Heho scenario. Flight examiners generally are interested in what the pilots 
were attending to and what was going through their minds, which would assist them in 
better understanding where to situate and what to define as the underlying issue. For 
example, a case is described and analyzed in Chapter 10 where the question is whether 
some issue was due to a pilot’s aviation knowledge, lack of situation awareness, or 
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problematic decision-making. Finding out requires work; and it requires the cooperation 
of flight examiner and pilot. It is this work that is at the heart of the present section. 


Articulating Forgetting and Lack of Knowledge or Awareness 


One of the issues flight examiners may be confronted with, especially in the case of pilots 
that they also rate as having problems, is the process of forgetting, not knowing about, 
or not having been aware of something on the part of the pilots. Flight examiners have 
access to these states only in and through the debriefing talk. Consider the following 
exchange that is part of the Heho debriefing scenario. 


Scenario 7.3 


01 PF :  Ithink I was not on the right heading, so I turned on zero three zero degrees 
to correct for the inbound, which is zero one zero. 

02 FE : Do you know how far off the inbound course you were when you made 
the left turn? 

03 PF : I think about fifteen or twenty degrees and then I corrected it the other 
way around. Like I turned right to intercept the inbound zero one zero. 
But I don’t recall exactly. I forgot. 


Whatever the meeting produces that actually can be used in constituting what has 
happened and what the pilots were aware of is the result of the joint work of pilots and 
flight examiners. In the course of such talk, pilots may realize that there were errors 
in the approach that they had not been aware of before, and they may list things like 
timing, initial approach speeds (139 knots), being worried about distance, and tracking 
in the wrong direction, so they had “to correct [the flight] the other way around”. 
That is, in the give-and-take exchanges with the flight examiners, descriptions come 
about that individual pilots might have never produced on their own. This additional 
material is the result of the generative aspect of dialogue, which yields more than the 
sum of what individual participants know and bring to the situation (Bakhtin 1984). 
Novelty emerges unpredictably from the internal dynamics of groups (e.g., Garud et 
al. 2015); and the dynamics of any group, as stated before, is endogenously produced. 
Flight examiners might offer up questions that inquire about specific aspects of the 
flight, which provide the pilot with opportunities for articulating and establishing the 
degree to which something is remembered and the quality thereof, the degree to which 
something is known, or what the (after-the-fact established) level of awareness was 
during the flight. A first occasion arises in Scenario 7.3 when the reply turn offers up an 
approximate measure of the aircraft’s deviation from the course and a final concluding 
assessment pertaining to the quality of remembering (turn 03). The PF’s talk, directed 
and intelligible to the flight examiner, indicates a lot of uncertainty, which is expressed 
in the use of modifiers (e.g., it was “about 15 or 20 degrees” or in the use of hedges (e.g., 
“I think”). The turn ends stating that the respondent “doesn’t recall exactly”, followed 
by a more definite “I forgot”. Such hedges are pervasive in some debriefing meetings. 
As shown above, flight examiners then may provide resources—e.g., by describing a 
particular aspect of the flight—to assist pilots in remembering. 
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Initiating Remembering 


If flight examiners eventually articulate some positive evaluative turn, they thereby 
indicate that the pilot has provided the correct or desired answer. For example, flight 
examiners explicitly ask at what stage the aircraft is to start descending. 


Scenario 7.4 


01 FE : Atwhat stage do you have to be fully configured on this NDB approach? 
02 PF : When I reach the altitude of five hundred feet. 
03 FE : But the ATC had reported to you IMC conditions? 


In this scenario, the expected answer includes (according to Air Bagan standard 
operating procedures), an approach track angle of 010+5° at an altitude of 500 ft in 
visual meteorological conditions and 1,000 ft in instrument meteorological conditions. 
The contrastive “but” of turn 03 marks that there is a problem with the preceding 
statement that constitutes the reply to the initiating query. Whereas the statement 
(turn 02) is correct in some sense, it does not take into account or state the contextual 
particulars: the altitude is 500 ft if there are visual meteorological conditions. As seen, 
the take up articulates a different set of conditions, IMC, where the required altitude 
differs from normal according to the standard operating procedures. Turn 03 therefore 
also provides an opportunity for the PF to remember the altitudes as function of the 
meteorological conditions. If the PF now clarifies the issue by adding the meteorological 
conditions, flight examiners would know that the knowledge of standard operating 
procedures is not the issue. The failure could be treated as the result of a situational 
oversight (perhaps a one-off). However, if the PF does not provide a differentiated 
reply, then this fact is usable as documentary evidence (see Chapter 2) of problems in 
the human factor of knowledge (aviation, aircraft, and standard operating procedures). 
In situations such as the present scenario, pilots have to realize that their actions do not 
conform to standard operating procedures. This requires both articulations of what has 
been done and what should have been done according to company policy. 

The question about when the aircraft has to be fully configured makes sense in a 
context where it had not been in the correct position, for this opens up the possibility 
of assessing whether the pilots know the stages in the configuration of the aircraft for 
this particular type of approach (NDB). During the scenario, the PF had announced, 
as he should have, that there are 540 feet to go to the minimum descent altitude. (The 
complete cockpit voice record, apparently appended to the original report of the Air 
Bagan crash, was not provided with its main body available for the present work.) As the 
official record shows—which would be available in the case of a debriefing tool—the 
aircraft was not established at the point where it should have been established given 
the heavy fog between aircraft and runway that the pilot in command had reported to 
ATC while approaching MDA (MAIB 2013). 

Throughout the scenario pilot’s attempts to respond to the queries, requests for 
elaboration also mark any preceding answer as insufficient or deficient: as not having 
provided what the initiation intended to elicit. A lot of repair work may be required, 
inviting replies that state the specific point at which something should have happened, 
such as the point at which the checklist ought to be complete. The incident report states 
that the crewmembers were experiencing a high workload during the approach, including 
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the work of configuring the aircraft, making radio calls, and conducting the checklists. 
In fact, the PF might have replied in turn 02 (Scenario 7.4), “Oh yea, that’s right, we 
were late in terms of configuring the aircraft and stabilizing it for the final descent. And 
this made us rush everything a bit”. In this instance, the very back-and-forth nature of 
the debriefing exchange leads to remembering. In the process, the PF might be brought 
to the point of acknowledging that the aircraft was both low in profile and late with 
respect to achieving configuration and completion of the checklist. Flight examiners 
interested in understanding what was in the mind of the pilots might now invite them 
to state what their thought processes were in those instances of the approach. Flight 
examiners thereby not only initiate queries but also constitute an evaluation of the 
foregoing as not having articulated the thought processes that they are interested in. 


Establishing the Actual Flight Track and the Associated Reasoning 


After initiating the process of remembering, an important aspect of debriefing in this 
scenario is the work of establishing the actual track and the associated reasoning (thought 
processes). As the testimony of the actual PF in the Heho incident shows, he described 
the aircraft as having been on track whereas the objective evidence shows that he had 
not been. For pilots in such a situation to be able to recognize and understand why 
things had gone wrong, they need to understand that they have been off course, how 
much, and where the aircraft has been tracking. In such situations, flight examiners often 
use the whiteboard to produce diagrams that provide special affordances because of 
their durability of drawings over the course of the remainder of the debriefing meeting 
generally and with respect to the exchanges concerning the event particularly. 


Scenario 7.5 


01 FE : What was your outbound track? ((Draws Figure 7.7a)) 

02 PF : Two two zero. 

03 FE : So two two zero. Are you aware where your needles were at the end of 
the outbound track? 

04 PF : Two two zero. Kind of. Or not? 

05 FE : You were on atwo two zero, but not aligned with the beacon. So you were 
off to the west. How would you allow for that when you turn inbound? 


Such exchanges establish (a) knowledge of what the outbound track should have 
been (available from the approach plate) and (b) the PF’s lack of awareness of the actual 
outbound track and the consequences that this would have on the subsequent stages 
of the flight if he had been aware of this deviation. In this scenario, flight examiners 
then asks the pilot whether he was aware of the needles on the bearing and automated 
direction finder (ADF, as shown Figure 7.4c). In taking up this issue, the contents of 
what the pilot said are problematized (turns 02, 04), which the pilot signals as happening 
in the querying “Or not?” at the end of the second reply. The next turn repeats and 
thereby confirms that the preceding reply was not just a statement about how the aircraft 
should have been tracking but also where the first officer thought he was tracking. That 
statement of tracking 220° is then contrasted with the statement that the actual track 
had been 220° but was not aligned with the ADF needle. 
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Figure 7.7 Successive stages of a drawing of the kind that flight examiners—alone or with the help 
of pilots—produce in cases such as the Heho flight scenario. 


In the course of the exchange, the drawing (Figure 7.7a) comes to record what for 
the meeting is to be the objective state of affairs. Here, the rendering of the published 
approach and the aircraft’s actual approach come to be contrasted: their difference 
is vivid, which would be further enhanced if flight examiners were using different 
colors. Thus, a first function of the drawing is the fact that it makes salient precisely the 
difference that the PF had not been aware of during the flight and during the time right 
up to this point in the debriefing meeting. There is a second function of the drawing 
in that the actual course of the aircraft during this part of the flight remains visible for 
the pilots while continuing to talk about the approach. There is a durability associated 
with the drawing that contrasts the ephemeral nature of the debriefing talk. The drawing 
saliently displays the deviation; and, in so doing, it is the very ground against which 
the subsequent talk occurs and that this talk needs to take into account. Indeed, the final 
turn of Scenario 7.5 invites an exchange over what the PF should have done had he 
been aware of the deviation at the location marked by the circled cross (Figure 7.7a). 
The contents of this talk offers the possibility of contributing to the flight examiner’s 
understanding of how to assess the PF, for if the latter replies appropriately, then the 
flight examiner has documentary evidence that the knowledge exists but that there 
existed a (temporary) lack of situation awareness possibly related to the high workload 
situation that the incident report also points out. 

In the following excerpt from the NDB debriefing scenario, the joint work 
establishes the absence of knowledge about how to re-join the correct track when the 
aircraft is outbound. That deviation, moving on a 220° track parallel to the published 
one is problematic but still allows for corrections, for the pilot always could have made 
corrections in the course of the base turn so that the aircraft eventually would have 
been aligned with the correct bearing (Figure 7.7b). Instead, the aircraft ended up at 
around the 25° radial (= 205°-180°). Once that part of the track is established in the 
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meeting, flight examiners would again ask pilots what they remember. Something like 
the following exchange might then be observed. 


Scenario 7.6 


01 FE : Sodo you remember what you did at this point? 

02 PF : Icontinued for a while and then tried to correct it by doing thirty more, 
like heading forty degrees to intercept with zero-ten inbound. 

03 FE : So you took a heading approximately through here ((draws the actual 
course taken from the circled cross up to the intercept, Figure 7.7b)). 

04 PF : Yea, that's about right. I guess. 

05 FE :  Sowhatwere you thinking while tracking through here? ((Gestures along 
actual track inside the teardrop.)) How did you feel? Were you comfortable 
to have everything under control? 

06 PF : Irealized I was off. And then I corrected to the right. But felt I was using 
too much of an angle to inbound. 

07 FE : So how did you feel about your workload? How did you feel about your 
management? 

08 PF : The workload was high. It all became a bit too much. 


Flight examiners tend to include the pilots in making the events present again, 
to have pilots narrate what has happened for the purpose of learning from it as far as 
they are capable given that they may have forgotten much of the situation and given 
that they might not have been aware of certain aspects of the flight in the first place. 
However, as there are pilots who do not remember, asking them to produce a narrative 
account does not get the meeting far enough for subsequently establishing what can be 
learned from the event and how to improve it. The uncertainty about the exact tracking 
is apparent from the {turn 03 | turn 04} exchange, where the constative statement of the 
actual course is taken up by and paired with hedges: “that's about right” and “I guess”. 
It is at this point that the exchange is important. There will then be many contributions 
on the part of the flight examiner to establish what has happened precisely enough so 
that the pilot can indeed learn to better deal with such situations in the future. This is 
quite apparent in this fragment, which ascertains where the aircraft was and how the PF 
was feeling. The flight examiner here asks about lived-experiencing rather than about 
some objective account, and the language explicitly asks for the affective dimensions 
of that experiencing (i.e., “How did you feel?”). Here, readers need to remember that 
the feelings articulated now in the debriefing meeting are not those that the PF actually 
has had. This is so because the PF is only talking about the feeling (and thinking) rather 
than actually having them; and, as theorized in Figure 6.2, it is not the original feeling 
that comes to be present again but whatever had been the content of experience viewed 
through the lens of everything else that has been happening since. At the same time, 
as known from “flash-back” phenomena, persons actually physically relive what they 
have lived before; they do relive affective aspects of an earlier situation. This form of 
reliving thus is of the same kind as reliving a particular movement that pushes the go- 
around button in the aircraft and that makes the same thumb or little finger move during 
the debriefing meeting where the current talk is about a missed approach. 
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This exchange establishes, therefore, the precise route that the aircraft actually 
had taken together with the contents of the PF’s conscious awareness (1.e., as it appears 
to him after the fact, during the meeting). In the early part, it was apparent that there 
was a divergence between the pilot’s account of where the aircraft was and where it 
actually had been. The latter was established in the course of the exchange, and includes 
the production of a diagram. As a result of this exchange, the situation awareness is 
determined to have been inappropriate. The sequence is used to establish that the pilot 
not only has been unaware of what the instruments were reading but also (and as a 
consequence of the first) where the aircraft was actually located geographically. Here, 
the diagram on the whiteboard functions as an important device. It is a developing 
representation of where the aircraft actually has been and where it should have been 
in the course of the approach. Perhaps more importantly, the diagram establishes what 
the factual matters are for the purposes of this meeting. Thus, for example, whatever 
the PF says following turn 03 in Scenario 7.6, it has to account for the fact that the 
aircraft was off course. This is already apparent in turn 02, where the PF states having 
taken a heading of 040°, whereas according to the original testimony, it had been on 
track at that point in time (see above). That is, one may ascribe a normative power 
to facts in that they have a future orientation that shapes the narrative, which has to 
take into account what actually has happened. Thus, the PF now states to have been 
off the course (turn 06) even though this was not part of the original story line. From 
the flight examiner perspective, it is possible at this point that the PF was not aware of 
having been off course; it is equally possible that the PF was becoming aware during 
the flight, as per the action of turning on 040° to intercept; and it is possible that he 
was aware and constructed the initial narrative for the purpose of presenting himself 
in a particular light. 


Teaching | Learning 


In the opening part of Chapter 5, the social psychologist Vygotsky is quoted with respect 
to the origin of higher psychological functions and personality. According to the cultural- 
historical approach to psychology, anything specifically human in human behavior was 
a social relation first becoming a function of the individual (without losing its social 
characteristics). As soon as the genesis of development is situated in social relations 
between people, it is no longer possible to separate teaching from learning (see also 
Chapter 9). Teaching implies learning as much as learning implies teaching. Indeed, in 
the educational psychology literature, Vygotsky’s Russian term obucenie has currency 
because it translates both teaching (instruction) and learning. Obucenie constitutes an 
irreducible unit, which usefully is expressed in English as “teaching | learning”; in other 
words, it manifests the unity/identity of teaching and learning (Roth and Jornet 2017b). 
The upshot of this position is that if flight examiner talk is not associated with pilot 
learning, then de facto no teaching (instructing) has taken place even though it might 
have been intended. The use of this term actually opens opportunities for analyses in 
that it orients investigators to the possibility that an institutionally designated teacher 
(instructor) learns while teaching and that an institutionally designated learner also 
teaches (contributes to teaching). Thus, for example, flight examiners become better at 
their job with each recurrent training and assessment session that they are conducting— 
as evidenced by the changes they and others observe as flight examiners gain experience 
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(see Chapter 2). We thus retain this unit for the present purposes, where teaching and 
learning occur simultaneously in the give-and-take of debriefing exchanges. 

At some point in debriefing meetings, flight examiners take the opportunity to 
initiate some form of teaching | learning exchange. This is so because producing an 
account of an experience would not yet be enough for acting differently in the future. 
In the aviation culture, members are aware of this, as can be seen from the common 
practice of asking for or marking as “the take-home message”. It is in this step that 
practitioners move from merely understanding the world (here their previous actions) 
to actually changing the world (their future actions). The preceding subsection shows 
how an experience relating to the final phase of the approach is discovered in and 
through joint work. But the diagram is still incomplete in that an account has not yet 
been provided of how the pilot flying could have rejoined the 010° course from the 
location marked in the drawing by a cross—and, therefore, an account of how the pilot 
should rejoin the correct inbound course in any future situation. There is therefore an 
opportunity for initiating teaching | learning so that the PF in the present debriefing 
scenario is afforded to learn what to do to get the aircraft into the correct position where 
it can be stabilized for conducting an appropriate approach—which here, because of 
the IMC, is an altitude of 1,000 feet. 

At the present point in the scenario, the diagram on the whiteboard includes the 
actual outbound track, the initial turn that the aircraft was to make until it would be at a 
90° angle with respect to the radial half way between 220° outbound and 010° inbound, 
the correct inbound track at 010°, and the actual track taken across the published inbound. 
The actual drawing may be treated as incomplete, as there is a gap between where the 
aircraft should have been until reaching base at the end of the teardrop-shaped course 
and the correct inbound course (Figure 7.7b). In the course of a teaching | learning 
segment, such gaps are filled in (see addition to diagram in Figure 7.7c). In fact, we 
may think of the gap itself as asking to be filled in, and, in so doing, affording the 
opportunity for teaching the first officer what should be done in that situation. That is, 
when we consider the debriefing meeting as a whole, after the initial additions have 
been completed (Figure 7.7b), it is in a particular state—not unlike the kitchen of short- 
order cooks (Agre and Horswill 1997). Any action can be thought of as calling for the 
transformation into another state closer to the outcome. If the in-order-to motive of 
debriefing includes learning in the form of a take-away message, then there is a definite 
goal for the transformation of the current state into the future state: the articulation of 
that course which the aircraft should have taken once the pilots realized that they were 
off course (at the circled cross, Figure 7.7). Flight examiners would therefore make an 
invitation for a teaching | learning sequence that would then respond to the gap (Figure 
7.7b) by closing it (Figure 7.7c). In such situations, flight examiners can be observed to 
initiate teaching | learning by formulating a rule of thumb. That is, they formulate that 
whatever follows is an instant of teaching, which would consist in the articulation of a 
heuristic for dealing with such flight situations. In the case of NDB approaches, flight 
examiners talk about the angles between ADF, heading, and desired inbound track; and 
they routinely can be seen drawing the aircraft or marking the angles in the diagram 
(Figure 7.7c). Or they might ask pilots to draw alternative routes, such as overshooting 
the inbound track to rejoin it from the other side. 

The diagrams are integral part of the teaching | learning segment for establishing 
how the pilots get the aircraft on the correct inbound track even when it has been off 
track, as in the present scenario. In the end, the course of the aircraft is plotted for 
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most of its course, with the important information of the deviation at the end of the 
outbound part, the location where it was after the turn, and, in bold the actual course 
by means of which the pilot wanted to correct for the deviation once he had become 
aware of it. With such diagrams, teaching | learning exchanges establish how the turn 
on base is to be effectuated so that the aircraft ends up on the final bearing (here 010°). 
The two turns to be taken and when they are to be taken are established in the joint 
teaching | learning work of flight examiners and pilots. If none of the pilots does, the 
flight examiner might remind them of a rule of thumb, whereby the ADF needle at the 
apex of the Heho NDB would be 15° off the desired approach of 010°; the second turn 
would then be conducted such that he ADF needle would be less than 5° off the 010° 
approach (Figure 7.7c). 


Articulating the Performance in Terms of Human Factors 


In many instances, flight examiners provide their assessments in the talking-head 
mode, telling the pilots where the main problems were in terms of the their familiar, 
company-specific human factors discourse—such as in the case described in Chapter 
6, where the flight examiner articulates situation awareness, management, workload, 
and the relationship between these factors. As shown in the final incident report of 
the Heho incident, situation awareness is a plausible factor that flight examiners and 
accident investigators invoke. A scenario of a typical assessment might then sound 
something like this: 


Scenario 7.7 


Throughout the exercise, you showed a lack of situation awareness. You did not 
note that you were on the wrong outbound track; and you realized too late that you 
failed to return on the correct inbound. A lot of things were done really late. This 
NDB is a high workload time. So I wanted to find out what was on your mind; I 
wanted to find out what your thought processes were. I am not so much concerned 
about the NDB approach itself. This is something that we can fix through some 
additional training. My biggest concern is that you continued the approach even 
though you were not configured in time. The corrective zero-forty course you took 
shows you realized being off-course. But you continued. Your decision-making was 
inappropriate, because it put you in the position of not being configured for landing; 
and this is a bad position to be in. 


In this evaluation of the pilot, situation awareness is marked to stand out as the 
most apparent deficiency. But it apparently is not the most crucial one. Above we see 
how flight examiners pursue finding out about what was going on pilots’ minds, the 
reasoning, inherently opening up the possibility to find out about the decision-making. 
We do see in the assessment paragraph that management is highlighted, here in terms 
of the workload. The NDB approach is characterized as highly demanding in terms 
of workload, which pilots can and should manage by accomplishing tasks sufficiently 
early to free up attentional resources for the case that something is going wrong. The 
clincher for the flight examiner in this scenario, as stated in the assessment paragraph, 
concerns decision-making: the pilot decided to continue the approach, and persisted in 
this even though the PF quite apparently had realized not being aligned with the correct 
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010° inbound. The aircraft was not set up in time for an appropriate final descent and, 
therefore, a missed approach should have been declared quite early on. 


THE WORK OF DEBRIEFING 


The assessment of pilots commonly is taken to be a measurement issue, with the 
attendant problem of making assessment more reliable and consistent (see Chapter 4). 
Considerable variation tends to be observed when multiple examiners assess the same 
performances; and investigations have shown that even extended (three-year) training 
episodes do little to improve the situation in any substantial manner. Subsequent 
research concerned with the cognition of flight examiners suggests that assessment is 
not a measurement but a categorization issue (see Chapter 3), which is well explained 
by a mathematical model based on fuzzy reasoning (see Chapter 4). All such studies 
are based on the same presupposition that the performance itself could be assessed—by 
means of whatever human factors model was used—simply by observing it. However, 
the assessment of pilots’ skills and thought processes may be approached differently 
by investigating cognitive activity without cognition, that is, by placing cognitive 
“phenomena within the give-and-take of social activity, rather than tracing them to 
individual or cognitive processes” (Lynch 2006). This approach is consistent with 
the stance that thinking, memory, decision-making, and other phenomena generally 
attributed to cognition can be made to appear in debriefing meetings. How this is so is 
described and explained in the preceding section. The particular focus is on the work 
of (a) debriefing that makes aspects of mind (including thinking, thought processes, 
reasoning) visible for the purposes of learning from what has happened and assessment 
and (b) constituting specific memorable experiences that stand out so that pilots can 
learn from it to improve their future performances. 


The Work of Making Mind Visible 


Psychological studies in aviation commonly investigate pilot performance in terms of 
skills and human factors that are supposed to underlie what pilots are thinking. This 
chapter proposes a different orientation by taking pilot cognition as the product of 
rather the input to the debriefing meeting. This change of focus is necessitated because 
flight examiners do not assess these human factors in the simulator and by some magic 
ability that allows them to see into pilots’ minds. Instead, debriefing meetings often 
are concerned with the production of cognition, mind, reasoning, or thinking. They 
do so after the fact, without any possibility of getting at the original event and lived- 
experiencing (see Figure 6.2). Here, the question therefore cannot be about the level 
of correspondence between (a) what is said about cognition, mind, reasoning, and 
thinking in the meeting and (b) the actual processes as they have happened during 
the flight event. Instead, the matters as constituted in the meetings pragmatically 
are taken as (reflections of) the actual states of affair. But it may indeed be better to 
leave these matters to the debriefing rather than having flight examiners base their 
assessment only on the observations in the flight simulator. The debriefing meetings 
can produce additional information not only additively (by including what the pilots 
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have to say) but also transactionally (by giving rise to emergent understandings because 
of the dialogical nature of verbal exchanges). The factual states, as shown in the case 
of the unfolding diagram, have normative character in that they must be taken into 
account in the narratives that describe and explain what has happened. Assessment 
and recommendations for training are based on the outcomes of the give-and-take 
exchanges that constitute the narratives and associated depictions on the whiteboard. 

Among other things, the analyses corroborate that “what people really think, know, 
intend, etc., are the actively managed interactional concerns of talk, text, and other 
expressive domains of social life, typically backdated, of course, to having occurred prior 
to the act” (Edwards 2006). The present chapter exemplifies the joint work required for 
understanding what has happened from the pilots’ and flight examiners’ perspectives for 
the purpose of understanding the assessment and the reasons on which the assessment 
is built. This work exists in the form of the give-and-take of the social relations. This 
is so even though, we might say paraphrasing Livingston (2008), the lived-work of 
solving the problem of finding out what the pilot was thinking is the work of devising 
and implementing, modifying, and planning future ways of searching and examining 
the pilot to discover the solution-relevant inter-relational details of the issue at hand. 

In debriefing, participants make accountably visible what pilots are thinking and 
how they are reasoning while flying the aircraft. The members to the setting engage 
in the sensuous work of “finding out what the pilot was thinking”, even though and 
especially if pilots have forgotten some of the key aspects of the flight from which they 
could have learned to improve their future performances. In a weaker sense of “social”, 
both flight examiners and pilots need to be part of that work. This is so because the 
former has more reliable access to the objective circumstances, where an aircraft was 
during simulation, and what the instruments read, whereas the latters’ thinking, invisible 
during but associated with the flight, is to be made visible. This thinking is to be made 
visible not in some generic way but mapped to the factual details of the particular flight. 

Thinking, for the praxeological purposes of mundane debriefing, is not a hidden 
mechanism with hidden variables that accounts for what pilots do. Instead, for the 
purpose of debriefing meetings, “what pilots are thinking” needs to come into existence 
so that it can be documented in assessment portfolios. These portfolios may (a) be 
used in support of particular assessment scores, (b) be employed in support of making 
recommendations for training and professional development, or (c) become a basis for 
pilots’ reflections on practice for the purpose of improving it. Those “deeper” events 
of interest to the cognitive sciences, if they do in fact exist, would not be helping out 
in the present situation because, unseen, they cannot serve as the evidentiary basis for 
taking any next steps in the training of pilots. That is, investigating how what pilots 
think or what form of cognition was associated with previous actions serves institutional 
purposes. This chapter exhibits how pilots’ thinking is “shaped by institutional 
requirements as to what is to count as recordable in the circumstances” (Antaki 2006), 
which here are the debriefing meetings and their consequences for the records, training, 
and pilot careers. 

In studying debriefing, investigators may seek to discover how the work of 
debriefing comes to be what it recognizably and identifiably is for flight examiners 
and pilots. That is, even if debriefing were to occur in some neutral place away from 
the simulator facility and some flight examiner or pilot were to accidentally open 
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the door, they would recognize what is going on. If they can do so then it is not only 
because of the content of the witnessed exchanges but also because of the ways in 
which participants manage the event; that is, they can do so because such meetings are 
endogenously organized and produced, making everything available required for pulling 
off and succeeding in this endeavor. That is, if participants and observers can see that a 
debriefing meeting is happening, then whatever it is that is seen has been made visible. It 
has been made visible so that the meeting recognizably becomes what it is. Specifically, 
in studying the work of “finding out what pilots were thinking”, investigators find out 
what makes “finding out what the a pilot was thinking” recognizably and identifiably 
the joint work of flight examiners and pilots. What is it that allows insiders to recognize 
that “figuring out what a pilot was thinking” is currently happening? Whatever it is that 
investigators can find, it will also be what makes flight examining utterly social and 
cultural, because it produces the visibility of its own reasoning. 

Prior to entering the debriefing room, flight examiners may gloss the impending 
meetings as “seeking to understand what a pilot was thinking” so that they can decide 
whether to pass or fail the pilot—e.g., here given the NDB approach scenario. During 
debriefing itself, flight examiners may repeatedly formulate the present purpose as 
“trying to understand the thought process (thinking) of the pilots”, thereby formulating 
for the purpose of their counterparts just what is happening in the then-and-thus of the 
meeting. They may explicitly state in the meeting that a pilot has failed the examination. 
That is, the thinking they want to understand is then exhibited for flight examiners to 
perceive, and therefore for the pilots as well. Of course, it is then also exhibited to the 
cognitive anthropologist directly or vicariously (with cameras) present in the meeting. 
That thinking, that display of thinking, is and always will be a social phenomenon 
in a strong sense. It requires the cooperation and collusion of flight examiners and 
examined pilots. 

It is remarkable that thinking is accessible and can be obtained in a debriefing 
meeting with nothing other than what the participants have at hand. In other contexts, 
complicated apparatuses of investigations and methodological specifications exist and 
are articulated for extracting from the data obtained what and how people are thinking. 
In the present context, no such apparatus exist and there is no special codified knowledge 
and practice that are taught and made available in specialized, certified institutions. 
But it is also unremarkable, for flight examiners do indeed come to the conclusion that 
they have found out and comprehended what pilots were thinking and how they were 
reasoning; and they have done so in the course of the unremarkable, mundane event 
that they call debriefing. 


The Work of Turning Inchoate Lived-Experiencing into An 
Experience 


This chapter also is concerned with investigating the work of re/discovering and 
constituting “an experience” in what has been an inchoate and ill-comprehended 
happening. We may usefully think of “an experience” in terms of the analogy with 
a story where the author tells both the objective conditions of some events and the 
internal, subjective feelings and private thoughts of the protagonists. It is from such 
stories that pilots can learn. 
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This NDB approach as flown by the Air Bagan pilots—though it likely would have 
resulted in a fail rating from every flight examiner participating in this research—would 
allow the pilots to learn for their future work, not allowing such an event to reoccur 
during their retraining and after returning to regular line-oriented work. As stated above, 
although the Air Bagan Flight W9011 crashed, deviations from the flight course of this 
kind do not inherently lead to the failure of a pilot or to the dismissal of a pilot in the 
case of failure. Instead, a low rating or retraining schedule may be the consequence. 
The flight examiner reasoning is based on whatever the debriefing meeting produced. 
In and by means of exchanges such as those described above, the participants jointly 
produce an account of what has happened in the way that this becomes apparent to the 
pilots. In analyzing the debriefing meeting it is possible to elaborate an approach to 
sensuous lived-experiencing that is consistent with the ways in which philosophers, 
sociologists, and psychologists have articulated the category (Dewey 2008; Schiitz 1932; 
Vygotskij 2001). The distillation of an experience from what has been inchoate lived- 
experiencing, that is, the reconstruction of it, only partially and incompletely recovers 
the transactional nature of cockpit life. For one, an experience covers the objectively 
available environment (e.g., how the aircraft actually moved with respect to ground, as 
might be seen on a radar), the practical actions of the subject, and how the situation is 
reflected intellectually and affectively in the pilot. An experience always has the quality 
of a story, with protagonists and plots, causes and effects, beginnings and endings, etc. 
The definition of the term makes intelligible how a failure such as the bungled NDB 
approach can become an experience: “courses of actions in which through successive 
deeds there runs a sense of growing meaning conserved and accumulating toward an 
end that is felt as accomplishment of a process” (Dewey 2008a). It is precisely the joint 
account—interlacing whatever the pilots’ sense had been with an intelligible, inherently 
shared account—that allows a “sense of growing meaning”. It allows this sense even 
though and precisely because the end result, now comprehended in objective terms, is 
“the accomplishment of a process”. 

The debriefing meetings provide pilots with opportunities to comprehend what had 
happened and why it had happened from an observer point of view; and these meetings 
generally provide opportunities for flight examiners to find out the pilots’ thinking 
(thought process) from a first-person perspective. This perspective is required for 
recommending specific training tailored to the needs to the pilots. Here “an experience” 
is understood as a category that includes (a) the objective, external environment, (b) 
persons acting in their environments as these appears in their perception, and (c) the 
intellectual and affective reflection in the persons. An experience always presents 
itself in narrative and definitive form, and it therefore has lost the inherently open, 
unfinished, in indeterminate form of lived-experiencing. Professional development—.e., 
the formation of new qualitatively new forms of rather than an incremental change in 
consciousness—may arise precisely when flight examiners and pilots, in their current 
lived-experiencing (E), constitute an experience (Ej) from materials that had been 
given in lived-experiencing (E, see Figure 6.2). In this chapter, debriefing meetings 
are investigated where, because of the intensity of the task, little if anything will have 
stood out from the experiential stream. This distinguishes the present situation from 
those discussed in the literature (e.g., Dewey 1934/2008), where the sense of having 
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had an experience (e.g., “that dinner”, “that evening”) immediately emerges. Here, in 
the case of aviation debriefing, the nature of the event—or, rather, a series of micro- 
events in the larger event of a bungled NDB approach—from which participants can 
learn has to be established first.? This, as shown above, requires the joint work of flight 
examiners and pilots and their representational tools (here the whiteboard). 

In the production of the account of what has happened, the participants contribute 
in ways that maintain the conditions for contributing in those ways: participants act 
such that the debriefing meeting recognizably remains a debriefing meeting. Participants 
do so even when the conditions change, such as when a shift is produced from a 
participant-centered to a flight examiner-centered way of organizing the current topic. 
When training departments decide implementing facilitated debriefing, then pilots, 
though they recognize something new is occurring, nevertheless competently participate 
in the joint work of making this new type of format succeed. The joint work of flight 
examiners and pilots includes the orientations that allow others to recognize the special 
role and contributions of each participant. Thus, even though pilots examined also may 
be flight examiners, their contributions will be such as to make their roles in the meeting 
recognizably that of an examined pilot rather than that of an examiner (see Chapter 5). 
That is, independent of their rank participants act in ways that produce the conditions 
of participating as examined pilot or flight examiner; and the pilots simultaneously are 
subject and subjected to the judgment of the examiner. 

In the analyses, the contributions of one representational technology are sketched. 
These analyses exemplify how the technology involved (whiteboard drawings) provides 
maps (of the aircraft, its states) that come to be related to, and constrain, the pilots’ verbal 
accounts of their lived-experiencing. Pilots thereby can transcend whatever sense and 
recall remained from their lived-experiencing towards the narrative constitution of an 
experience from which they may learn. The differential role of the artifacts present in 
and used during the debriefings is further examined in Chapter 8. Here it may suffice 
pointing out that being-in-the-world, absorbed in the ongoing activity, characterizes 
the immediacy of the original flying situations related to the motive of pilots everyday 
work activity. Thus, the immediacy and the perspective from within the subject still 
characterize what is lived and witnessed even when analyzed together by those who 
inhabit their world. As critical sociologists and psychologists alike recognize, going 
beyond that immediate lived-experiencing, transcending its immediacy, requires 
comprehending the situation as a whole, environment and the individual, and the 
respective histories thereof (Holzkamp 1983; Smith 1999). This, here, is achieved by 
means of the representational technology that presents the objective, outside (third- 
person) perspective. In this, the present work constitutes a view of an experience as 
a social phenomenon in the strong sense. Thus, an experience is not “mediated” by 
social relations, but rather exists in and through the unfolding social relations that alone 
provide for its intelligibility. An experience, in the sense taken here, is not just subjective. 


? There is a problem in Dewey’s articulation, because in the course of the evening, “that dinner” can 
have very different referents should something like a rampage shooting occur. Any that dinner can 
be an experience only when it has been completed—as those diners surviving the November 2015 
restaurant attacks in Paris would readily agree. 
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¢ Debriefing and individual or collective reflection on praxis presupposes the presence 


of past events, which requires representation or the presence of the past. 


e Cockpit tools and artifacts or iconic representations thereof support remembering 
(“flash backs”) and practical understanding. These tools and artifacts also encourage 
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and facilitate pilots to make movement sequences (body, actions, talk) that they 
have produced earlier. This leads to an intensification of the past in the present. 


+ Abstract models (representations) tend to come in the verbal and associated 
graphical modes. Whereas action models represent sequences of action types, and, 
therefore, relate to and describe what pilots do in the aircraft, conceptual models 
are less useful and accessible to them serving instead flight examiners and training 
managers for decontextualized assessment information. 


tudies that are taking a closer look at what happens in debriefing tend to highlight 

that there are different foci that may arise in such meetings, including those related 

to the practices themselves (technical issues of the field and non-technical issues 
pertaining to the relation between pilots) and those related to the interface of humans 
and computers. In aviation debriefing interface issues are rarely observed; and the 
meetings are almost entirely focused on both technical and non-technical performance 
aspects. Where appropriate, flight examiners introduce relevant performance dimensions 
as topics of discussion during debriefing. The topics fall into the technical realm (i.e., 
aircraft knowledge) when pilots, for example, do not know what an engine instrument 
should read when the propellers of a failed engine do not align with the flight path; the 
topics fall into the non-technical dimension of situational awareness when pilots failed 
to be aware of instrument readings. The purpose of the debriefing meeting includes 
finding out the root cause of a performance problem, because any advice for professional 
development or recommendations for training will differ accordingly. 

Existing studies of interaction often investigate debriefing in formal educational 
settings where learners, participating directly or vicariously in (medical, dental) 
operations, come to perceive relevant objects in the manner that seasoned practitioners 
already do. In one study, additional visualization was made possible when researchers 
attached a camera to a surgical microscope during a dental procedure (Rystedt et al. 
2013). The camera offered a close-up into the procedures that the dentist carried out 
inside the mouth of the patient. Because the students were provided with a view through 
the eyes of the dentist, they could make sense of the instructors’ clinical in situ reasoning. 
This provided them with the context required for understanding dental procedures. 
Even those studies that had been investigating the interactions—coding dimensions 
such as the nature of issues (e.g., technical versus non-technical human factors)—end 
up making suggestions for further research that (a) improves understanding debriefing 
practices in detail, (b) addresses the lack of knowledge about debriefing processes, and 
(c) investigates how the division of labor shapes processes and outcomes of debriefing 
meetings. 

Chapter 7 provides insights into the tremendous work required to assist pilots in 
remembering specifically what has happened during the preceding four-hour flight 
simulator session. When they exit from the simulator, many pilots initially describe their 
preceding experience as a blur. Whatever detail returns to their conscious awareness 
does so during the exchanges with flight examiners while talking about the different 
exercises that they have completed before in the simulator. During these debriefing 
sessions, pilots frequently express surprise about having or not having said or done 
something while flying. These details tend to arise in the give-and-take with flight 
examiners, who also draw on their notebooks or recollections, or when pilots see 
themselves in the videotapes that some companies use to record the simulator sessions 


Differential Effects of Tools and Artifacts in Debriefings 225 


(as part of the debriefing tool set up). It is frequently only when situations are made 
present again—when they are presented or represented in some form—that from the 
preceding stream of inchoate lived-experiencing some fragments have come to stand 
out as parts of experientially closed wholes (e.g., events). It is from these remembered 
event segments that pilots learn to perform in the future when they find themselves in 
non-normal situations of the kind that they do not or seldom encounter during their 
everyday flying. They nevertheless have to be prepared flying such situations just in 
case they actually occur. That is, the only way to prepare for flying expertly under non- 
normal conditions is to live through these in simulated situations where there is no risk 
to aircraft, passengers, and pilots. But if such situations appear as a blur, if there is little 
standing out from the stream of lived-experiencing, then how can pilot learn “from their 
experiences” and thereby become ready for the unexpected? The existing research on 
debriefing generally and on debriefing in simulation-based learning specifically tends 
to be silent (Fanning and Gaba 2007). Central to the concerns of this chapter is the 
question, “What role do tools and artifacts play in the constitution of a (memorable) 
experience so that what has happened may indeed become an object of analysis and 
the basis for learning and improvement of practice?” 

This chapter is concerned with the role and function of tools in the social practice 
of debriefing, the purpose of which is to make present again aspects of what has been 
sensuous lived-experiencing and thereby constitute an experience in the Deweyan sense. 
Here pilots learn while and as a result of debriefing past performances with the flight 
examiners, who have observed these performances. In this regard, these debriefing 
sessions constitute forms of formative assessment, where prior performance is not 
simply assessed in a dispassionate manner but rather oriented toward the emergence 
of insights. Throughout the debriefing sessions, the recall and constitution of specific 
flight-relevant events is one of the central problems of debriefing activity. This chapter 
shows that recall is not just achieved verbally, but involves bringing the past to life in 
and through a variety of performative and material means, including different types 
of tools and artifacts. 


A THEORY OF TOOLS IN DEBRIEFING 


In Chapter 1, several systemic approaches are presented for understanding complex 
human activities involving tools and artifacts: joint cognitive system and cultural- 
historical activity theory. The former focuses on the interactions of separate (elementally 
conceived) cognitive agents, who pass information back and forth between one another. 
Differences are conceived in terms of different interpretations of the agents. The latter 
approach has advantages in that it conceives of complex situations, such as those in a 
debriefing room, from a transactional viewpoint that not only recognizes the constitutive 
nature of the whole and its parts but also has time and change as internal characteristics. 
Cultural-historical activity theory treats material and ideal aspects of the situation 
as different manifestations of a single underlying unity/identity of agents and their 
environments. In Chapter 1, an archetypical triangular representation is used to make 
thematic the different parts of the activity of the movement of an aircraft from airport 
A to airport B. An equivalent representation may be used to make thematic all those 
aspects involved in a cultural-historical perspective on debriefing activity (Figure 6.1). 

Chapter 7 articulates the tremendous cognitive work required to produce first 
remembering and memory of the past events, which then become the object of reflection 
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and analysis for the purpose of assessing pilots and producing a plan, if required, for 
further professional learning. Whatever is remembered in the course of the debriefing 
meeting itself becomes material—i.e., part of the object (Figure 6.1)—to be transformed 
and integrated in the final outcome. From the perspective of cultural-historical activity 
theory the whole debriefing meeting constitutes the unit of analysis; it cannot be further 
broken down into independent elements (e.g., pilots + flight examiner) that are then 
brought into interaction. To be a unit in activity theoretic sense, debriefing has to have 
all the characteristics of society. Transformation is inherent in the theory because every 
temporal segment of debriefing is part of the larger movement from the beginning, when 
pilots often have forgotten, to the full articulation of what has happened, completed 
assessment, and plan for further improvement. This change is made explicit in the model 
generally and in its thematization of the movement from object > outcome specifically 
(Figure 6.1). That outcome, before being actually achieved, is the motive of the activity. 
In Chapter 7, a division of labor is observed whereby pilots and flight examiners engage 
in the joint (cognitive) work to produce an account that leads to understanding precisely 
because, as this occurs in novels, it includes subjective first-person perspectives and 
objective third-person perspectives. The accounts also include the airlines’ assessment 
discourses; and these discourses have many standard and standardized aspects when 
there is an explicit model of performance and performance assessment. 

In the present chapter, another aspect of the model is highlighted: the role and 
function of the existing tools in the debriefing process and on its outcomes. As the 
model shows, these tools are an integral aspect of the production process (Figure 6.1, 
apex). The outcomes are functions of the tools used; and the tools shape the production 
processes, as tool-use shapes any next stage of the object in its transformation toward 
the outcome, which again is the object at the next instant that is transformed by means 
of the tool. The object is like a snowball, the making of which is shaped by some tool, 
which then is applied to an object that already has been shaped before. When the tool is 
a human factors model—such as the one present in the debriefing meetings of several 
airlines in the form ofa laminated printout—then its use will have shaped the outcomes 
of the meeting. That is, tools shape the transformation of the object—which, in its new 
form, is subject again to the extant tools—at any one point in the movement from initial 
object to the ultimate outcome. One way in which tools shape the meeting has been 
described and theorized in Chapter 5. The tools and artifacts also are part of, related 
to, and constitute the division of labor, a fact that is inherent in the model (Figure 6.1). 
The relationship of tools and actions during the different parts of recurrent training and 
assessment is made thematic in Table 8.1. 

The table makes apparent the different positioning of flight examiners and pilots 
with respect to the events in the simulator, where flight examiners configure the 
simulation based on the flight test syllabus but pilots actually fly the scenarios. The two 
parties therefore come to be differently positioned with respect to the observations and 
the use of recording tools: the flight examiners are situated outside the action, observing 
much as spectators and journalists observe a sports event, whereas the pilots live the 
situations from within, with body and mind on the field. Here we are concerned with 
the cognitive affordances of different types of tools and artifacts to debriefing, to the 
production of memory, to the production of third-person perspectives, to reflection 
on action by comparing actual with referent (ideal) performances, and, ultimately, to 
different kinds of understanding and learning. 


Differential Effects of Tools and Artifacts in Debriefings 
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Table 8.1 Selected Tools and Actions in Simulator Training, Debriefing, and Post-debriefing Situations. 


























Phase Tools and Functions Flight Examiners Pilots 
Simulation | Assessment plan, flight test | Configure simulation 
syllabus! 
Full-motion simulator Perform simulation 
Recording tools Observe Experience (with body 
e Notebook and mind) 
* Debriefing tool 
Evaluation tools Evaluate 
e Performance model e Task performance 
e Rating forms 
Debriefing | Feedback tools Describe situations from | Describe situations 
* Debriefing tool observer viewpoint from agent viewpoint 
(notebook) (experience, thinking) 
Replay video Watch situation from 
objective viewpoint 
» Flight artifacts (QRM) Describe referent Remember + compare 
performance actual to referent 
performance 
e Action model Describe referent Remember + compare 
(F-DODAR) performance actual to referent 
performance 
e Assessment model Provide assessment Self-assess 
States improvement, Describe “take-away” 
remediation plan 
Post- Documentation tools Document ratings, 
debriefing describe situations, justify 
ratings 














'The assessment plan and syllabus is the outcome of the work of the training manager/s, who thereby 
comes to influence all debriefing meetings, the products of which return to the office of the training 
manager, who is also involved in providing for any retraining schedule. 


Tools and artifacts roughly may be classified in this way. Some things, because 
of their iconic relation to the simulator flight events, make possible and promote 
remembering work and develop the practical (rather than theoretical) understanding 
of flying an aircraft. These include the debriefing tool, charts, and manuals from the 
cockpit (quick reference handbook). Other things have a more abstract, non-iconic 
relation to the flight events and are used to conceptualize actions or underlying skills. 


TOOLS AND ARTIFACTS FOR REMEMBERING AND PRACTICAL 
UNDERSTANDING 


There exist a variety of tools and artifacts that are used in the debriefing room, including 
a debriefing tool (videotape and instrumentation), flight artifacts (e.g., charts, manuals, 
handbooks), and cockpit posters. Videotapes of flight events and iconic representations 
of instruments and actuators (control column, flap lever, etc.) make certain aspects of 
flight events present again in the way that these have occurred. In the case of videotapes, 
pilots are afforded to see themselves in the way that flight examiners have seen them 
(see drawing in Scenario 1.1); and pilots hear what they actually have said. This supports 
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both remembering (e.g., as in “Oh yea, it is coming back”) and a reflexive stance toward 
what they have done (e.g., as in “I can’t believe I said that”). When instruments are 
shown in the same way that they had been available to pilots during the flight, these 
represented instruments come with specific affordances. They can be read as they 
have been (or should have been) read and, in this, they support remembering; in some 
instances, they support pilots in discovering that they had missed certain information 
(see Chapter 10). Cockpit-related artifacts and tools, such as airport approach charts 
and manuals such as the quick reference handbook with the checklists, also are used 
in the debriefing room. They therefore do not represent something else but they stand 
for themselves. Pilots reading a checklist in the debriefing room are confronted with 
the same text that they had been confronted with in the cockpit. They may actually 
discover during debriefing that there had been a problem with their reading while 
flying, such as when they discover having missed an item. Thus, the presence of the 
identical tool or artifact tends to assist pilots in remembering, leading them to have 
“flash-back” experiences in which the past becomes present again and, therefore, object 
for reflection and learning. In the first two of the following subsections, the functions 
of the debriefing tool and of cockpit artifacts (charts and quick reference handbook) are 
discussed in a representative manner to stand for those things in the debriefing room 
that present themselves or have iconic relation with objects and events in the simulator. 
The cockpit poster has a special role in that it makes present in an iconic manner the 
material part of the cockpit. 


Debriefing Tool 


The debriefing tool makes available different modes of representing the original flight 
in the simulator (Figure 8.1). In addition to the video of the cockpit as seen from the 
flight examiner sitting behind the two pilots (top left, Figure 8.1), the debriefing tool 
offers the possibility of representing key flight instruments such as the electronic 
altitude director indicator, electronic flight instrument system, the engine indicators 
(e.g., center left and center), and aspects of the flight management system (waypoints 
and track). Also available are iconic representations of relevant actuators, such as the 
control column, power levers, and flap levers. Other options include representations of 
the vertical and horizontal tracking with respect to the ideal descent path and inbound 
track similar to the information available in Figure 7.4 (e.g., posted track with protection 
zone and actual track). Flight examiners use a panel (often placed on the bottom right 
of the monitor) to select relevant flight episodes; and they control the selection and 
size of the representations for the purposes at hand. For example, they may show the 
video only covering the entire monitor, thereby displaying it much larger than shown 
here (Figure 8.1). 

Video is possibly the most widely used means to make present again past events 
for the purpose of reflection on practice in learning science research (e.g., Goldman et 
al. 2007); video also constitutes the principle multimedia means in debriefing sessions. 
Integrated with a number of flight instruments and representations recorded during the 
flight simulation, videotapes constitute an important resource for the process of making 
the past present again. When the videotapes are played, pilots see themselves in the 
event of interest. In some situations, flight examiners merely play the videotape and 
then initiate a conversation about what was seen and heard. More important may be 
those instances when flight examiners reproduce the movements seen on the display. 
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Engine indicators Aircraft elevation 
Cockpit Electronic flight ¿ Steering column 


instrument system 


Electronic altitude 
director indicator 


(pilot & side view) 
(as FE sees it) : 

: Power levers 
: Flap levers 





View (what pilots see or Waypoints and track Management tools 
aircraft from behind) 


Figure 8.1 The configuration of the debriefing tool includes, next to the video of the cockpit during the 
simulator sessions and a view of the aircraft from behind or from the cockpit, also a variety of iconic 
representations of flight instruments and representations of actuators (e.g., steering column, levers) 
related to the flight event shown in the video. 


These flight examiner movements are important because they cue pilots to look for 
and see an equivalent movement in the video. It is precisely the relation between the 
performance in the debriefing room and what can be seen in the video that makes these 
movements stand out as the relevant phenomena. Thus, for example, if a pilot in the 
video is leaning with the upper body in a particular way during a left turn, this leaning 
already comes to stand out when the flight examiner verbally links it to the movement 
of riding a motorcycle; and the movement and body configuration is especially 
highlighted when flight examiners move as if making a left turn on a motorcycle. The 
flight examiner movements therefore constitute aids in picking out from a complex 
and non-self-explicating display precisely those aspects that are important for the 
present purposes. Previous research on demonstrations has shown that some viewers 
of live demonstrations may see movement where others do not see movement; and 
the explanations observers subsequently produced were geared towards explaining 
whatever they had seen (Roth et al. 1997). This suggests that flight examiners should 
not take for granted that pilots will see whatever they want them to see. Ambiguous 
optical phenomena, such as Rubin’s vase that also can be seen as two faces, provide 
a good analogy for situations that lead to very different perceptions. This is important 
because when instructors provide explanations, the audiences integrate whatever is 
said with what they have seen. That is, if audience members have seen movement 
(or a vase in Rubin’s case), the explanation is heard as explaining movement; and if 
they have not seen movement (or the two faces in Rubin’s case), the same explanation 
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explains the absence of movement. The debriefing tool provides flight examiners with 
the opportunities to align all participants with respect to what is seen and heard in the 
videotape played. 

In such a situation, the video affords alerting pilots to their past actions and to the 
different aspects on the instrument panel that they should be aware of during the flight. 
If pilots in such situations had not been aware of having done something (e.g., leaning 
like a motor cyclist during a turn), they would now be confronted with their actions as 
seen by another person. That is, video presents pilots with an outside perspective on 
their own actions and movements and with representations of aspects of the cockpit. 
The tool therefore affords the interlacing of two perspectives. Understanding consists 
of the fusion of the external viewpoint of the generalized Other and the viewpoint of 
the Self; all human knowledge exists in that interlaced form (Husserl 1973). These two 
perspectives therefore are constitutive of and required for human knowing. Observing 
others already involves the interlacing of the perception of the Self (present to itself) 
here and the apperception of another Self (alter ego) as if the Self were there. In the 
case of pilots viewing themselves, the interlacing involves another doubling because 
that other Self, now seen in the videotape, also is their own Self in the way it had been 
some time earlier. In seeing the videotape, pilots come to see themselves in the way that 
they normally see others (other selves). Just as understanding another person requires 
an external viewpoint on the Self, understanding the Self requires the viewpoint of 
another on the Self. Out of this interlacing evolves understanding of the Self, arising 
from the successful understanding of the Self as another (Ricceur 1992). In addition, 
because of the reciprocity of Self and Other, “such understanding uncovers my own 
psychic life .. . by bringing new features into prominence, makes it fruitful for new 
associations” (Husserl 1973). 

There is more to the debriefing tool because it also presents the pilots with the 
instruments in the same way that these were accessible in the simulator. The pilots 
see in the here-and-now of debriefing approximately what they have or could have 
perceived there-and-then, through the lens of what has happened since (see Figure 
6.2); in fact, because they are no longer in the same situation involving, for example, 
a heavy workload, pilots may now see something that they have not been aware of 
while flying and the process of seeing will have changed to what it perhaps should 
have been. In such instances, the instrument may have been present to them in some 
way (A) without appearing in conscious awareness (W,(A)) and without having been 
made present internally (V,(A)) by means of some representation. Reading the attitude 
indicator or the airspeed indicator requires certain dispositions and movements. This 
comes with opportunities, for when the (eye) movements are the same, episodic recall 
is stimulated and pilots are afforded to recognize what they have perceived and felt 
before. Consider the following scenario of the type that might occur after the pilots 
have failed the NDB approach to Heho (Chapter 7). Flight examiners, after having 
discussed with the pilots in detail the different problems that have occurred, may then 
invite the latter to watch themselves as captured on camera. 


Scenario 8.1 


The flight examiner initiates the viewing episode by stating that it is important doing 
so for the purpose of learning how to better fly NDB approaches. Flight examiners 
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would do so unless there existed reasons for a loss of the pilot license. The purpose 
for looking at the video is thereby formulated—not for the purpose of assessment but 
for the purpose of learning from what can be seen as happening so that performance 
can be improved in the simulator on the following day. Flight examiners configure the 
debriefing tool to feature those aspects relevant to the situation. In the NDB scenario, 
this may include the video, electronic flight instruments (featuring magnetic compass, 
heading, and automatic direction finder [ADF]), descent path, and horizontal path. 

The debriefing tool provides for opportunities to make present again the actual 
course of events in the cockpit. In the scenario, the aircraft had just made the turn at 
the end of the outbound track and, as could be seen from the instruments, had not 
aligned itself with the needle, which in fact was therefore not on 010° as it would 
have been had the aircraft be in the right place (position 4, Figure 7.4; ADF position 
Figure 7.6d). This was followed by corrections, an account of which might have 
already been produced earlier in the meeting (Chapter 7). As the video runs, the pilots 
observe the instruments again. They do so not in a mode of an inner reproduction 
of a former external perception but by the inner awareness of the same external 
thing—.e., in the same mode in which they previously (could have) related to the 
instrument. Simultaneously, the video continues to play. The flight examiner provides 
a running commentary for the purpose of highlighting something, such as pointing 
pilots to the position of the ADF, which was not opposite to the heading; pilots 
state what they were doing and thinking, such as when the heading was changed 
by 90° to bring about the first turn away from the initial 220° outbound course. 
The video is stopped to provide opportunities for discussing the different stages of 
the flight. For example, the Air Bagan standard operating procedures state that in 
cases of IMC, the aircraft has to be configured by the time it descends to 1,000 ft 
above ground. Flight examiners therefore may ask, “At 1,000 feet above ground, 
what is your configuration now?” The pilots might reply stating: “It has to be Flap 
42° and before landing checklist complete”. Given that the aircraft was not aligned 
with within the margins of the required 010° inbound, flight examiners would then 
highlight this aspect: “Where are you tracking now?” 


In this way, a complete mapping of the actual occurrence—instruments and 
actuators from the pilots’ viewpoint and from the flight examiners’ viewpoints—and 
the desired flight path and actions can be achieved. Pilots thereby come to re-live the 
past (£,) in some form in the present (as £7); and what they re-live will be relative to 
what they ought to have done, which also has to be present in some form, for example, 
in a performance or a narrative. In so doing, pilots gain a critical perspective onto their 
own actions that are made present in the dual, interlaced views from the perspective 
of Self and Other. Even though “the ego and the alter ego are always and necessarily 
given in an original pairing” (Husserl 1973), the video affords a literal view on the (past, 
different) Self as an Other.! In this movement, the subjective sense that is re-lived in 
the “flash back” comes to be interlaced with the objective sense related to everything 
perceived as outside the Self. 


' Over the course of nearly 30 years of research, I frequently witnessed participants being astonished 
and even embarrassed when they see themselves (on video) in the manner others see them all of the 
time. 
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Cockpit Tools and Artifacts 


Artifacts from the simulator session are used in many debriefing sessions, including 
onboard manuals, on-route charts, and approach plates that contain the landing 
information (Table 5.1). Typical artifacts from the cockpit include the flight crew 
operating manual (FCOM), the quick reference handbook (QRH), the landing speed 
chart, and the takeoff data sheet. Present in both simulator and debriefing settings, 
they provide pilots with opportunities to act with tools; and such availability of the 
identical tools is integral to and constitutive of what comes to be produced as an 
experience. Special perceptual opportunities arise because the presence of these entities 
affords being internally aware of something externally present rather than having to 
reproduce internally what has been an external perception (cf. Husserl 1928). In the 
here-and-now of the debriefing meeting, these tools and artifacts are part of sensuous 
lived-experiencing; and this lived-experiencing may lead to a re-living and recognition 
of the past, which thereby comes to be both past (it is recognized as having occurred 
before) and present (it is cognized here and now). That is, the cockpit tools and artifacts 
function as aids in making present (V,(A)) again some entity A, especially in sessions 
where there is no debriefing tool available. The same process of making present therefore 
occurs at time ¢, and ¢, in the form of VA) and V,"(4) involving the same entity A 
again; this, as the observations show, is easier for pilots than trying a reproduction (R) 
from memory of the original presentation, that is, R(V°(A)). Analogical in function to 
those artifacts are the instruments displayed by means of the debriefing tool. These 
instruments, though clearly representations, are associated with the same orienting and 
reading practices that the actual situations afforded while pilots were flying at work or 
in the simulator. The phenomenon already has been described more than two centuries 
ago in the context of how humans recognize and remember shapes of figures that they 
have felt before (Maine de Biran 1841). Thus, when the hand takes or tries to take the 
same disposition with respect to an object previously felt (a statue, a globe), then we 
find ourselves in the same active state as before and we perceive. Even if the object 
is absent, the object is touched by thought so to speak. In aviation debriefing, these 
instruments that appear on the debriefing tool therefore function as proxies for the real 
thing rather than representations thereof. This is so because the perceptual processes 
involved in reading the torque gauge as displaying 90% will be the same in the simulator 
and in the debriefing room. This role of presentations is exemplified in the following 
episode in which aviation charts are used in the debriefing room. 

Charts are an integral part of flying. In the debriefing sessions, participants use or 
refer to charts in presentational form—.e., the official charts from the civil aviation 
authority or the charts appearing on the debriefing tool; or they refer to them as 
representations—i.e., as charts printed out from the simulator session or charts drawn 
on the whiteboard. Flight examiners tend to use a pen, index finger, or cursor to follow 
the path of the aircraft on the chart while narrating or discussing with pilots relevant 
flight-related events. In the following scenario related to the airport close to this author’s 
home, the participants orient to a chart to discuss the events during the approach to 
the specific airport with the runway marked by its orientation (RW27, 1.e., 270° on the 
compass) (Figure 8.2). This chart is part of an approach plate that provides a pilot with 
all information required for conducting a safe approach to and landing at the airport. 
These provide a pilot a (a) top “birds eye” view of the airport and surrounding terrain; 
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(b) safe track an aircraft can follow to avoid high terrain and other air traffic, such as 
departing aircraft; (c) vertical profile that assists a pilot in determining appropriate 
times to descend during an approach; (d) variety of radio frequencies for both ground- 
based navigation equipment and air traffic control communication; and (e) specific 
notes applicable to the airport. Even small airports may have as many as ten approach 
plates, with large city airports having in excess of twenty plates. Each pilot must carry 
a suite of approach plates for all possible airports they may use on a particular segment 
for normal and possible emergency situations (i.e., having to decide deviating to an 
alternate airport). Whereas these charts traditionally are held in a binder, some pilots 
and examiners access them now in electronic form on tablets. 

In this scenario, the flight examiner discusses with the pilots the particulars of the 
approach they flew in the simulator. Talking about the arrival of the aircraft, the flight 
examiner indexes the waypoints (e.g., UKNUP, LAFFO), positions between these 
waypoints, and the altitude information provided. The flight examiner points out that 
setting the required (altitude) minima early on decreases the pilots’ workload on the 
LAFFO/TUGUK section, which is the flight segment just prior to making the final 
descent. During an approach, the pilots set the altitude that the aircraft has to attain in 
the event that the landing will have to be aborted (e.g., because of poor visibility). If 
the pilots were to leave setting this altitude to a later point during the approach, they 
would thereby increase their workload at that point in time. 


Elk Lake 
Beaver Lake 





Figure 8.2 Schematic representation of some of the information that appears on an RNAV approach 
plate into Runway 27 Victoria, British Columbia (CY YJ). The stars represent waypoints, the names of 
which appear in capital letters. Elevations (feet) appear above and the distances (nm) below the heading 
for each segment between two waypoints. 
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Scenario 8.2 


01 FE : Sowecame in two six nine from out here ((D/SCO)) to UKNUP ((points 
to UKNUP, Figure 8.2)). Then we came around ((points to LAFFO star, 
Figure 8.2)) and onto the approach. When we started doing R/NAV in 
our airline, the standards department initially specified that the minima 
could not be set until the aircraft actually is established inbound. Now our 
company procedures allow pilots to set the minima here ((points to 2200 
between UKNUP and LAFFO, Figure 8.2)). This is the beginning of the 
T-transition ((points to T-transition at LAFFO, Figure 8.2)). Our airline 
changed the procedure to reduce the workload around there ((points to 
LAFFO/TUGUK, Figure 8.2)). 


02 P :  Thisis exactly what happened to us. It was just enough workload to make 
use a hundred feet too high. 
03 FE : Thisis why we changed the procedure. Now you have enough protection, 


because your approach already is established when you cross the IAF. 


As the aircraft approaches an airport, there is a critical point at which pilots have 
to make the decision whether to land or whether to announce a missed approach leading 
to a go-around procedure. This point is given in terms of the minimal descent altitude. 
This altitude, which in this scenario the aircraft reaches a little over 1 nautical mile prior 
to the airport, has to be set in the advisory display unit. The instruction here pertains to 
setting the minimum between UKNUP and LAFFO rather than between LAFFO and 
TUGUK (where the final approach way point, FAWP, is situated), when the pilots are 
busy getting the aircraft ready for the first part of the landing procedure. 

In such instances (i.e., Scenario 8.2), flight-relevant materials are used in a 
retrospective manner, where the participants produce an account of what happened 
earlier in the simulator, why it has happened, and, where pertinent, what should be 
happening in the future. As seen in the scenario fragment, the chart itself affords a 
process of presentation rather than a representation: the chart stands for itself. Reading 
it requires the same underlying cognitive process whether it is used on the aircraft, in 
the simulator, or in the debriefing session. In terms of the flight, such artifacts serve 
as plans in two ways (Suchman 2007): retrospectively what the aircraft and cockpit 
have done and prospectively orienting the pilots to what they will have to do in future 
situations. These two orientations are associated with two different forms of motives: 
because motives are part of a posteriori accounts that take events as facts that can be 
explained in determinative manner and in-order-to motives orient actors to the future 
but do not cause their behavior. The chart in this instance contained what the pilots had 
used to plan their arrival. That is, at the beginning of the flight, the map was projectively 
organizing what would be done. During the debriefing session, it is used as part of the 
fact-constituting narrative that makes present again that segment of the flight currently 
under discussion. The chart is presentation and representation simultaneously, because 
the pilots do what they do when they read the map and they use it to make present again 
that flight organized by means of the map. Critical phases of the flight in terms of the 
non-technical aspects of performance are mapped against the trajectory of the aircraft. 
This is the case when flight examiners point to the location on the flight path where the 
workload was high, and where they gesturally act out changes in the specific aspects, 
such as where the initial approach fix (IAF) lies. These gestures are not arbitrary but 
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act out precisely those movements—in time and space—that pilots use while changing 
the setting of the minima when crossing the IAF. The flight examiner, in initiating the 
discussion of the approach might point to the LAFFO/TUGUK part of the map to make 
salient that the pilots ought to set their screens to alternate modes so that they would 
have maximum information available. 





Controlled Flight into Terrain Near Palmerston North 


The following information is contained in the final report, including the transcription 
of the cockpit voice recorder (TAIC 1995). 


Event Description 


On June 9, 1995, a DHC-8-102 (Dash 8) around 9:22 a.m. local time collided 
with terrain in a mountain range about 7.5 nm (14 km) east of Palmerston 
North (New Zealand). The aircraft had been flown to join the 14 nm DME arc 
and then turned to the right to intercept the published final approach track of 
250° to the Palmerston North VOR; this turn occurred about 13 DME from the 
runway. At that point the aircraft was still above the 3.1° descent angle (5% 
gradient) that the standard operating procedures at the time specified to be flown. 
However, the aircraft then began losing altitude rapidly and, by 9.5 DME, was 
below the hard lower altitude limits (3,000 ft). When the aircraft made contact 
with ground at about 7.5 DME from the runway, it was more than 1,000 ft 
below the 2,500-ft lower limit indicated in the approach plate. Four individuals 
(3 passengers, 1 crewmember) died; and 12 individuals were seriously injured. 

The captain, who was PF at the time, was a seasoned pilot with 7,765 hours on a 
total of four types, with 273 hours on the Dash 8. The first officer, too, was a seasoned 
pilot, with 6,460 hours (341 hours on the Dash 8) and rated for five aircraft types. 

In the approach, at just over 11 DME, the PM made an “on profile” call, but the 
PF did not correct the excessive rate of descent. About 15 seconds later, when the 
aircraft was 300 ft below the advisory profile but still 300 ft above the lower limit, 
the PF noticed a problem with the undercarriage system, asked the PM to grab the 
QRH but to whip through the first (non compulsory) items and said that he “will 
keep an eye on the aeroplane” while the PM was to go through the checklist. While 
the PM was going through the checklist, PF called area control (ACC); and it was 
in the course of the associated pauses that the PM missed two checklist items (see 
below). A few seconds later, the PF caught the error. The accident report suggests that 
this was a good cue for the PF concerning the desirability to abandon the approach 
so that the landing gear problem could be sorted out. At that time, a flight attendant 
opened the cockpit door to say that she had noted a problem with the undercarriage 
system. The pilots continued attempting to sort out the QRH checklist items when the 
aircraft’s electronic voice (AEV) called out “terrain, whoop whoop pull-up, whoop 
whoop pull up”. Less than five seconds later, the aircraft collided with the ground. 
There is some evidence that the PF started up-pitching the aircraft nose but there 
was no significant increase in engine torque required for climbing. 
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The accident report notes the following shortcomings in human performance, 
which an aviation psychologist had characterized as “attentional slips, memory 
lapses, and mistakes”. The captain did not apply sufficient power to intercept and 
maintain the approach profile; and he failed setting appropriate power to regain the 
normal approach path after the aircraft had descended through it. The PM incorrectly 
calculated the required altitude at 10 DME, and the PF’s reply “check” was treated as 
a confirmation rather than request to verify the calculation. The PM did not correctly 
track the QRH items after the landing gear failure was identified. Not only did the 
PF fail to monitor the descent path, he also did not address the absence of altitude 
monitoring calls. 

The following two subsections focus on two aspects of the flight that are 
discussed in Chapter 8 (the role of flight artifacts, here the quick reference handbook, 
QRH, in debriefing meetings) and Chapter 9 (social relations as genetic origins of 
higher functions and pilot learning exemplified in an instance of moving the power 
lever during a go-around procedure). 


Missing Quick Reference Handbook Checklist Items 


At 9:20:09, the PF asks for gear down. The PM notes 25 seconds later that the 
extension is not locked and announces alternate landing gear, to which the PF 
replies by instructing the former to get the QRH. At that point, the exchange with 
the flight attendant occurs. The following transcription begins just after the PM has 
replied to the flight attendant that they know about the problem and after the flight 
attendant has acknowledged the reply by saying “thank you”. (The right arrows 
mark the place where the PM skips items while going through the checklist, part of 
which is shown in Figure 8.3.) 


Excerpt from the Cockpit Voice Recorder (TAIC 1995, with information added) 


01 PM: Landing gear inop, landing gear malfunction, alternate gear 
eighteen, oh right ((checklist 18)), alternate gear extension, 
approach and landing checklist, pressurization ((first item in 
checklist 18)). 

02 PF : Oh just skip her down to the actual applicable stuff. 

03 PM : Landing data, altimeters, tanks, belt smoking, okay airspeed 
below a hundred and forty knots and landing gear inhibit switch 
inhibit ((Figure 8.3, second line)). 


04 PF : Okay, and it’s one forty ((Figure 8.3, first line)). 
05 PM : Landing gear selector... is down ((Figure 8.3, third line)). 
06 PF 2 MGS, 


> 07 PM: Landing gear alternate release door fully open, which it is (Figure 
8.3, fourth line)). 
08 PF : ((Zo ACC)) and Ansett seven zero three established finals at 
Palmerston North. 
> 09 PM: Yes thanks, and insert. 














10 


11 
=> 2 
13 
14 


15 
16 
17 
18 


ACC: 


BE 


PM : 


BE 


PM : 


BE 


PM : 
AEV: 
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Ansett seven zero three that’s understood, and contact Palmerston 
Tower one two zero six. 

one two zero six, thanks. 

Insert this handle (horn). ((Figure 8.3, sentence case entry.)) 
It’s noted. 

Insert handle at, till, oh yeah, and operate until main gear locks 
((Figure 8.3, sentence case entry.)), actually, nose gear. 

You're supposed to pull the handle . . . (laugh). 

Yes, it’s go it actually after that, yes that’s pulled, here we go. 
terrain, whoop whoop pull-up, whoop whoop pull up. 

(Sound of impact) 


LANDING GEAR MALFUNCTION ALTERNATE GEAR 


EXTENSION 








SCAIRSPEED RE ee ae mea eee aaa 140 KTS MAX 
= 5/SIINHIBITES WATER Ee en sssta-eeacetenree ener mner en mans INHIBIT 
eilt/ GISELE CTORIEE VER elo DOWN 
+ L/G ALTERNATE RELEASE DOOR ................2..4. OPEN FULLY 


+ MAIN GEAR RELEASE HANDLE .............. PULL FULLY DOWN 


+ L/G ALTERNATE RELEASE DOOR ................2.... OPEN FULLY 


Insert pump handle and operate until main landing gear locks 
down (LEFT & RIGHT green lights ON & L DOOR & RIGHT 


DOOR amber lights ON & movement becomes stiff). 


& LEAVE OPEN 


& LEAVE OPEN 








Figure 8.3 The figure typifies the appearance of checklists in quick reference handbooks that are 
used in the cockpit, simulator, and debriefing. Here, information from the accident involving a Dash 
8 near Palmerston North (New Zealand) is used for exemplifying purposes. 


Pilot’s Response to the GPWS: Power Up, Pitch Up 


Based on the flight data recorder (FDR), there was no significant increase in engine 
torque following the GPWD warning. The FDR provides evidence of an up elevator 
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input yielding a maximum angle of 6 degrees. The aircraft pitch angle increases 

to a maximum of 8 degrees nose up. These recorded parameters suggest that the 
PF at the time did indeed initiate a pull-up after the GPWS warning. The standard 
operating procedure used (available on board in the Flight Operations Manual) 
specifies that a GPWS warning requires setting the power leaver to “Go-around 
power” and choosing a pitch attitude for the best rate of climb. Go-around power is 
less than the maximum available power. The accident report states, however, that 
in situations such as a “hard” GPWS warning, maximum power should be set as 
quickly as possible and practical together with a sufficient nose-up angle to maximize 
the climb angle. Although Ansett Australia policies emphasized that the aircraft be 
pitched to 15—20 degrees nose up, the chief pilot responsible for the Dash 8 aircraft 
did not advocate such angles to avoid risking engine stall. 











The understanding tools also include the manuals that pilots have available in 
both the cockpit and the debriefing meetings. Just as airport charts that are used in both 
situations, these manuals do not constitute representations but are artifacts that stand for 
themselves; the artifacts are used and may appear in consciousness rather than being 
available through internal representations (Husserl 1928). They are present-to-hand 
in one as in the other situation, requiring, for example, practices of reading that carry 
across the two situations. But there are differences, for when pilots use these tools in 
the debriefing room, they may actually discover (i.e., become aware) that they had 
missed an item; and they may recover in their memories the reason for such a forgetting. 
Missing an item may have (or may be associated with) dire consequences. One such 
instance occurred in the controlled flight into terrain of an Ansett New Zealand DHC-8 
near Palmerston North, where the PM missed two items from a checklist for landing 
gear malfunction and seconds later the aircraft collided with the terrain. (In this case, 
subsequent recommendations by the Civil Aviation Authority UK, 2006, suggest that the 
design of the checklist was not optimal, referring to exactly this sequence and offering 
improvements that would increase the probability of correct checklist use.) Between 12 
nm DME to the airport threshold when the malfunction was identified and the instant 
of reading and missing checklist items around 7.5 nm DME, the aircraft had descended 
1,185 feet below the procedurally defined glide path and then made contact with terrain. 
The accident report explicitly notes that the captain was “not paying sufficient attention 
to the aircraft’s flight path while assisting the First Officer with the application of the 
QRH ‘Alternate Gear Extension’ checklist” (TAIC 1995, emphasis added). The captain 
was deemed to have acted appropriately when assigning the QRH-related actions to 
the first officer while he was going to continue flying the aircraft. However, the report 
concluded: “the intention, while appropriate was not followed. Had the situation it 
created been handled as intended, the undercarriage system abnormality which occurred 
on ZK-NEY should not have led to a collision with the terrain” (TAIC 1995) 

An example of the cultural practices and cognitive work involved in the situation 
is given in Scenario 8.3, a debriefing situation related to a landing gear release door 
problem. The appropriate checklist is that from the Ansett New Zealand Flight 703 
incidence (Figure 8.3). In this scenario, the pilots debrief an event similar to what 
happened just prior to the instant when the aircraft collided with terrain while bound 
for Palmerston North, New Zealand (see box). 


Scenario 8.3? 


01 


02 


03 
04 


05 


06 


07 


08 
09 


FE: 


PF: 


PM: 


FE: 


PM: 


PF: 


FE: 


PF: 
FE: 
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So take a look here. We had a 
failure in the landing gear. Run 
down through the checklist. We 
got down to here. [We went, 
“Landing gear alternate release 
door”]. 

((oriented to PM)) And you did 
that? 

I did that. 


[You did that and then what 
happened at about that time]? 


[I remember something with the 
ACC]. We were talking to ACC 
about there. 


Yes, with the ACC. 


That’s right. [Why do you 
think you missed], the main 
gear release handle? What did 
you do in the middle of the 
checklist? Do you remember? 


No, I don’t. 


[You waited for the captain and 
then?], 
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LANDING 


MALFUNCTION ALTERNATE GEAR 


EXTENSION 







* AIRSPEED ...... NEI MAA 140 KTS MAX 








+ L/G INHIBIT SWITCHED, nene INHIBIT 
+ L/G SELECTOR LEVER DOWN 
+ L/G ALTERNATE RELEASEDOOR .... OPEN FULLY 


& LEAVE OPEN 

+ MAIN GEAR RELEASE HANDLE .............. PULL FULLY DOWN 
- L/G ALTERNATE RELEASE DOOR... OPEN FULLY 
& LEAVE OPEN 

Insert pump handle and operate until main landing gear locks 
down (LEFT & RIGHT green lights ON & L DOOR & RIGHT 


DOOR amber lights ON & movement becomes stiff). 








LANDING GEAR MALFUNCTION ALTERNATE GEAR 


EXTENSION 


AIRE so peek a un ata iale ett 140 KTS MAX 







© L/G INHIBIT SWITCH. css ca cn caca ca tacă esa cai INHIBIT 













“L/G SELECTOR LEVER sasssscsccscstsssissasescceassaetesaassaanice 
+ L/G ALTERNATE RELI .., OPEN FULLY 
& LEAVE OPEN 
+ MAIN GEAR HANDLE .............. PULL FULLY DOWN 
+ L/G ALTE RAII SE DOOR .....0... eee OPEN FULLY 
& LEAVE OPEN 
erate until main landing gear locks 
ghts ON & L DOOR & RIGHT 


becomes stiff). 


LANDING GEAR MALFUNCTION ALTERNATE GEAR 
EXTENSION 





‘AIRSPEED ¡ora pă iio and 140 KTS MAX 
* LIS INHIBIT SWITCH sistiese tisanes ana sinaaa cae INHIBIT 
«1/6 SELECTOR LEVER serieei nada rasa oară a a . DOWN 

RNATE RELEASE DOOR... OPEN FULLY 


& LEAVE OPEN 


+ MAIN GEAR RELEASE HANDLE .............. PULL FULLY DOWN 


ALTERNATE RELEASE DOOR OPEN FULLY 
& LEAVE OPEN 


Insert pump handie and operate until main landing gear locks 
down (LEFT & RIGHT green lights ON & L DOOR & RIGHT 


DOOR amber lights ON & movement becomes stiff). 








2 The shaded parts of the text correspond to the visual on the immediate right. 
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The checklist from the quick reference manual here is a tool for describing, 
remembering, and analyzing a past performance without requiring flight examiners to 
retell the situation, necessary in such cases where the pilots forget the details of the event 
(e.g., turn 08). Flight examiners invite the production of an account by pointing to a 
particular item, which, in becoming part of the accented visible salient in consciousness, 
marks the point where the pilots thereby are invited to begin their accounting procedure. 
In this scenario, both pilots indicate remembering that they have done the item. It is at 
that point that the pilot monitoring moves forward, points to the checklist just above the 
bleed button, and, using a hedge of uncertainty (“something”) articulates remembering 
an issue with the call to the area control (ACC). That is, the checklist itself provides 
the item that the pilot now articulates as a description of the past event. The checklist 
functions as an external memory. When the pilots see the item, they remember it. The 
flight examiner acknowledges what has been said, but then seeks an explanation for 
the omission to deal with the main gear release handle (turn 07). A further direction 
toward something that happened while the checklist was done is provided followed by 
an explicit query whether pilots remembered. The reply is negative: the pilot remarks to 
have forgotten the event. The flight examiner then supplies the sought-for description: 
The checklist had been stopped to call ACC for the purpose of announcing that the 
aircraft was established on finals. 

Pointing does not exist in itself, in the movement of a hand, finger, or pen. Rather, 
pointing involves the relationship between a movement or configuration that is seen to 
be pointing at something; it is the something pointed at that makes the movement or 
configuration a pointing gesture. But whenever a movement or configuration is seen as 
pointing, its object becomes part of the accented visible. In the scenario, hand and pen 
movements are directed toward items in the QRH. These movements become significant 
because there is something at their end (or in prolongation thereof), an object that is 
pointed at. On the part of the recipient, pointing exists in finding the relation between 
a movement and an object. If there is trouble, recipients might say something like, 
“What is it that you want me to see?” or “What are you pointing at?” On the part of the 
person communicating something, the hand movement (pointing gesture) is motivated 
by something outside the person. 

This scenario exemplifies situation where the artifact becomes an organizing tool 
in the production of an account. Repeated pointing to relevant items affords locating 
the crucial point of the event account around the checklist item that had been omitted 
during the flight situation (turns 01, 04, and 07). In affording pilots to read the items 
again, a resource for remembering the event is provided when they had been reading 
precisely this text on this page. Consciousness of the item triggers the memory of the 
consciousness onboard the flight at the time when the same item was in the consciousness 
of the pilots: The making present of a perceptual object and the reproduction of the 
perception of the object yield the same phenomena (Husserl 1928). In the scenario 
there is indeed partial recall occurring in the sense that what happened was associated 
with a particular phase and aspect of the flight: Recall requires a representation, but 
this representation had not existed in the original event. As it happens frequently, the 
precise details therefore remain inaccessible to the pilots. It is in such situations that the 
access to the debriefing tool with its capability of replaying the situation clearly comes 
to the fore. In the manner shown below in the context of the (T-, F-) DODAR process, 
attending to the past performance while gazing at the checklist aids in identifying 
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what went wrong and, as apparent from the videotape, the identification of the reasons 
for deviating from the recommended completion of the checklist before attending to 
other things. 


Cockpit Poster 


In debriefing meetings, (static) iconic representations of cockpit posters with 
photographic realism are used quite frequently, in the present investigations 38% of 
all meetings involved them (Table 5.1). These posters include depictions of instrument 
and overhead panels, the pedestal (between pilots, containing power and flap levers), 
and the lateral panels. Cockpit posters are nearly identical with the representations that 
can be found throughout some manufacturers’ manuals for normal procedures (such 
manuals are intended as flight crew training manuals). In some manuals, the flows of 
actions (kinetic melodies) associated with different stages of the flight are indicated by 
means of arrows (not unlike the flows that can be seen in Figure 1.7). The cockpit poster, 
used during debriefing of widely different situations, does not contain such arrows. It 
tends to be used for the purpose of locating instruments (by means of deictic, pointing 
gestures) and to exhibit the sequence and flow of operations (kinetic melody). The poster 
makes the cockpit present in iconic form. Flight examiners walk or reach to the poster 
to point to a relevant instrument or actuator (e.g., flap lever). They may move through a 
relevant kinetic melody as if they were sitting in the actual aircraft, acting out a complete 
procedure from the standard operating procedure manual. In the following scenario, 
the flight examiner emphasizes a particular movement sequence, which is made part 
of the accented visible by means of the acronym PAC [power-attitude-configuration] 
(see below). Turning to the cockpit poster, the flight examiner says: 


Scenario 8.4 


P-A-C—Power, attitude, configuration. This means that you first check the power 
((points to the torque gauge on the instrument panel)). Number one engine, maximum 
takeoff power? ((Movement of finger as if pushing maximum takeoff power button.)) 
Yes. Now configuration: Are the flaps ((looks and points towards where flap levers 
would be located)) at ten degrees and ((gaze moves upward to gear lever)) the gear 
up? No, I am at eight degrees. So now, what does my next call have to be? 


Here, rather than talking about the sequence for extracting the aircraft from an 
engine fire at take off situation, the flight examiner performs the sequence of operations 
in front of the cockpit poster. The sequence thereby comes to be represented but in a 
perceptually situated rather than an abstract manner. The scene appears in the form of 
a little drama against the props rather than being presented in the verbal modality. In 
the scenario, the flight examiner moves through a performance that is to serve as the 
norm in the pilots’ future experiences to which the P-A-C sequence pertains. In other 
instances, pilots may walk to the cockpit poster and show where they looked and what 
they have done during a particular flight event. In such session fragments, therefore, 
pilots exhibit dispositions, orientations, and movements required to look for getting 
a particular work aspect done. The poster makes available the relevant aspects of the 
cockpit in a photorealistic manner so that it can be used and referred to even when a 
pilot does not immediately name something or remember some operational sequence. 
For the recipient (flight examiner), the aspect does not have to be represented but 
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already presents itself. Because of the kinetic qualities of their knowing, pilots may 
not (immediately) recall some relevant aspect of a flow and the instrument associated 
with it. Rather than having to produce knowledge about the flight event, they actually 
produce knowledge of flying as soon as a kinetic melody associated with the event is 
triggered. Chapter 9 takes a closer look into such an event where a go-around procedure 
is required; the chapter shows how it may be taught when a debriefing tool is available, 
and how what is a social relation between people (flight examiner and pilot) later 
shows up as a higher psychological function (î.€., the appropriate behavior immediately 
following the declaration of a missed approach initiating the go-around procedure). 


Affordances of Remembering and Understanding Tools 


This section considers some aspects of the differential functioning of tools and artifacts 
used in the production of an experiential account. There is a difference between those 
tools and artifacts that appear in the presentational mode (e.g., the instruments made 
available by the debriefing tool, the charts and manuals) and the representations (e.g., 
maps drawn on the whiteboard or appearing on the debriefing tool display), as seen 
in the debriefing scenario of Chapter 7. The former are deeply implicated in what the 
aircraft actually does while it is on its approach, whereas the latter is an a posteriori 
mapping external to the movement. The former take part in the actions of the pilots, 
and, in so doing, become themselves reconfigured in the process, for example, as the 
aircraft is moved to a different flight path. The same distinction holds between the 
altimeter and vertical speed indicator, on the one hand, and the map also available on 
the debriefing tool concerning the position (as seen from the side) of the aircraft along 
its flight path (similar to what appears in Figure 7.4), on the other hand. Whereas in 
the first type of cases, perceptions and reproduction of perceptions are at play, in the 
second type of cases a new and different form of representation is at work. The display 
of the debriefing tool makes jointly available (a) the first-person perspective that pilots 
have had and make present again in the joint work with the flight examiner and (b) the 
third-person perspectives in a representational space of how the aircraft was tracking 
(horizontally, vertically). That is, the two representations permit the production of an 
account in which the different aspects of an experience are combined: the objective 
environment and what was available to the pilots in that situation. 

In sessions where the debriefing tool is available, participants may directly refer 
to what one or the other instrument displayed during the flight and now displays again 
in exactly the same manner. That instrument is made present again. Because of the 
identity of the presentification of an external object and the inner reproduction of a 
(past) external perception (Husserl 1928), pilots are not required trying to remember 
that object. As participants are gazing at the instruments in the debriefing meeting, 
they see (and may see again) what these have displayed in the simulator at that time; 
and these are now displaying again what was or could have been seen in the past (see 
Chapter 10 for a detailed look at how instruments are and can be used in debriefing). 
The instruments thereby are available to the reading practices that also unfold or should 
have unfolded during the flight event. The participants not only read what presents itself 
but also know where to go to find the relevant information when they need it (Agre and 
Horswill 1997). The presence of the instruments featuring what they had featured in 
the simulator session then affords specific instruction on a particular issue. 
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The cockpit poster is a photo-realistic representation. It is present in all debriefing 
rooms, where in some situations it provides affordances similar to those observed in 
the case of the debriefing tool. It constitutes a representation of the instrument and 
overhead panels, the pedestal (between pilots, e.g., containing power and flap levers), 
and the lateral panels. These cockpit posters were used and referred to, for example, 
when a pilot could not name some part or cannot remember some operational sequence. 
Standing in front of the poster, pilots or flight examiner may point to such parts or enact 
a movement sequence over and about the representation. In this way, the cockpit poster 
triggers the making present again of a kinetic melody associated with the situation 
in the debriefing narrative, such as when participants point to the instruments on the 
debriefing tool monitor. 

The debriefing tool orients pilots to the very instrument display they would have 
seen during the examination. When flight examiners move the cursor next to or over an 
instrument (e.g., transcriptions in Chapter 10), they thereby provide resources for the 
instrument to become a salient part of the discussion: pointing accents the visible. In any 
setting of the cursor an additional fat bar eventually emerges from its tail, highlighting 
the instrument even further and thus instructing pilots to attend in a particular way (e.g., 
Fragment 10.3). In this way, pilots are invited to relive the situation but now with the 
opportunity to do so as originally seen and to see the instrument from aback, as the 
theoretician (reflective practitioner) might see it. However much the original (inchoate) 
lived-experiencing is present, it is necessarily intertwined with the pilots’ perceptions 
of themselves; and the view is not on the pristine event as it has been but through 
the cumulative lens of everything else that has happened since (see Figure 6.2). The 
interlacing of inside and outside perspectives is furthered when flight examiners select 
a mapping of the aircraft’s actual horizontal and vertical tracking against the inbound 
track with +5° markers (Figure 7.4a) and the prescribed 5.2% descent (see Figure 7.4b). 
This allows flight examiners to precisely locate the aircraft (e.g., saying “we are here 
now” while moving the cursor to the position of the aircraft in the representation). In 
the Heho debriefing scenario (Chapter 7), this affords the salience of the movement 
across the inbound track and the triangle marking the +5° allowance on either side. 
That is, whereas the previous instance of pointing orients pilots to the instruments 
perceivable during the flight, this time they are referred to a third-person view of the 
flight in single representational tool. What is offered up with the map of the aircraft 
tracking with respect to where it should have been no longer is how something looks, 
but how the aircraft moves when its position is marked out in some grid (Harper and 
Hughes 1993). The grid is an exogenous format where the aircraft comes to occupy a 
certain terrain in a certain way. It is a way of seeing and a technology that can be used 
to hold agents accountable for their actions (Suchman 1993). What the participants in 
the Heho debriefing scenario see, at that time in the debriefing room, is the fact that 
the aircraft is tracking across, rather than getting into and remaining within, the safe 
space that the standard operating procedures call for (Figure 7.4a). 








Kinetic Melodies Constitute Kinesthetic (Corporeal) Memory 


Iconic representational tools and cockpit artifacts afford the redeployment of hand/arm 
and body movements that are characteristic of flying an aircraft with instruments and 
actuators (e.g., control column or power levers) in their specific place. It allows for a 
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different kind of remembering process: a re-producing, recapitulative memory as distinct 
from a non-recapitulative memory (Husserl 1928). In recapitulative memory, duration 
is again at work so that “what was given just now” and therefore is part of immediate 
retention “reveals itself to be identical with the remembered”. When in debriefing pilots 
account for what they have done while flying in the simulator, they produce those same 
movements that are part of the cockpit- and work-related kinetic melodies. In such cases, 
the distinction between past and present (sensuous lived-experiencing) comes to be 
washed out, because the same forms of hand/arm and body movements and orientations 
take place (i.e., act 4). The memory is of a recapitulative kind, with all the phases and 
levels of the original lived-experiencing. The cognition (awareness W,(A)) is not exactly 
the same, though, for the productions in the debriefing meeting and awareness W'(4), 
are associated with in-order-to motives different than those in the aircraft where there 
was either no explicit awareness (i.e., merely act A) or an awareness WA). Indeed, 
because the events in the simulator are part of examination and assessment, cognitive 
events are not identical to those that take place during regular line operations. These 
differences are clearly addressed in cultural-historical activity theory (compare Figure 
1.9 with Figure 6.1). But even in the face of these differences, the investigation exhibits 
the apparent affordances of the tools and artifacts in the process of making the past 
relive in the present. Pilots tremendously appreciate opportunities of getting involved 
in the making present of what has happened; and the artifacts and tools lead to a more 
vivid presence of the past. In such situations, they again live and undergo certain kinetic 
events. This notion captures the idea that each aspect of life has its own overall, particular 
rhythmic movement leading to an unrepeated quality permeating lived-experiencing 
as a whole, including sounds and their intensities. 

The notion of kinetic melody (Chapter 1) captures that dynamic aspect of life 
generally and of the performance of a cockpit specifically. Kinetic melodies are not 
brain events but, “in a living experiential sense”, are “in-the-flesh dynamic patterns of 
movement that are initiated—and run off” (Sheets-Johnstone 2009). Capturing aspects 
of the original setting, kinetic melodies therefore are similar to a form of situationally 
organized bodily memory. The recurrent movement of kinetic melodies constitutes a 
form of memory that does not require representations, because the movement presents 
itself: Any act A is present as itself rather than in represented form V(A). These acts are 
the manifestations of elementary perceptions and therefore may serve to make present 
the same ideas that were associated with them in the past (Henry 2000; Maine de Biran 
1859). The memory ofa sensation is the sensation itself in the process of beginning to be. 
External objects appear in lived-experiencing where and as they are rather than within 
the consciousness of the person (as W(4), V(4), or W(V(A)). These appearances are 
part of a sense-constituting contexture characteristic of the world. The present chapter 
shows that the use of artifacts in debriefing meetings that are also present in the original 
event, facilitate the bodily presentification of the past, which thereby is made available 
as object to be discussed and reflected on. 

Presence plays an important role in learning from cockpit simulator events 
because the salience of former perceptions and lived-experiencing are a function of 
present perceptions and lived-experiencing, themselves related to the actions to be 
accomplished in the future. The ““movement of memory at work”. . . carries memory 
back so to speak into a region of presence similar to that of perception” (Ricceur 2004). 
Whereas a mental schema (representation) alone is but a sketch in the consciousness 
of the person, fully felt muscular sensations that come with the actual movement 
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and affective shading, bring color and life to the remembered event. Color and life 
appear to have the same function as the vividness that characterizes what stands out 
in an experience that was derived from and is related to the original inchoate lived- 
experiencing. The corporeal playing out of the past is an important aspect of debriefing, 
because “the body acts as a crucial site for commemoration” (Middleton and Brown 
2005), making kinetic melodies key phenomena in the constitution of an experience. 
It is in and through the bodily involvement that an experience, constituted in the form 
of a verbal account (î.€., a form of mimesis,), is (materially) preserved for future praxis 
(i.e., a form of mimesis,) modified by what has happened in the debriefing room. More 
strongly stated, the sensuous movements of the body constitute immemorial memory 
because it remembers, without requiring the crutch of “mediation”, past movements 
and the kinestheses that come with them. This is consistent with the cognitive analysis 
of how a cockpit calculates, remembers, and forgets speeds suggests kinetic melodies 
to be like multi-instrument performances, which, in the context of flying, consist of 
an irreducible joint cognitive system of (automated) pilots, instruments, and control 
devices (Hutchins 1995b; Roth et al. 2015). This articulation takes into account the 
ways in which experience has been theorized, that is, as the unity/identity of person 
and environment and its intellectual, practical, and affective coloring. 

In debriefing meetings, flight examiners and pilots contribute to the presentification 
of past experience (e.g., Ey) in and constitutive of present experience (E), as well as 
all other intervening experiences (Figure 6.2). The debriefing meeting itself appears 
in lived-experiencing (E), relating to the simulator as a job interview relates to job 
performance. The more kinetic melodies are present, the closer the relationship between 
the two forms of lived-experiencing. In the same way as the interview, the debriefing 
“may be mechanical, consisting of sets of questions, the replies to which perfunctorily 
settle the matter” (Dewey 2008). But this would not amount to an experience. The 
situation is an exercise in bookkeeping—which is what happens to be the case in those 
airlines where debriefing is used to communicate the results of an assessment. If the 
debriefing sessions are completely reduced to the production of conceptual accounts of 
what happened, there is little gain for the pilots in terms of professional development. As 
the popular adage of things working in theory but not in real life shows, understanding 
something on paper or in the debriefing room does not have to mean change in and 
of practical behavior. But, like the job interview, debriefing sessions may also lead to 
interplays that are memorable. This happens when the interplay projects participants 
“imaginatively into the work to be done and judges the fitness by the way in which the 
elements of the scene assemble and either clash or fit together” (Dewey 2008). 

In debriefing meetings, memorable (an) experiences are constituted based on 
earlier occurring, inchoate lived-experiencing associated with and arising from 
absorbed coping. The process of doing an act (aspect of practice) again is opposed 
to mental memory of represented acts and knowledge, because the doing-again 
transcends consciousness and expression (representation). In practice, people do not 
tend to memorize the past: They perform the past, which comes to be re-lived. What is 
learned through bodily engagement is not what pilots have but is something that they 
are. Corporeal memory therefore plays a special role in the transition from the past 
to the future. This is so because in corporeal memory, the body is event and the seat 
of lived-experiencing, tying together space and time in acting. The first dissociation 
between space and time comes with the association of memories and places where 
events have taken place. The things remembered (e.g., instrument readings, cockpit 
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artifacts) are tied to these places, which in turn are constituted by these things. Things 
and memory become reminders, that is, places where the corporeal memory brings to 
life again what otherwise might be subject to forgetting. Thus, through the memory 
of things in space pilots are enabled to recover the sense of spatiality associated with 
their practical actions on the job and in the simulator (see below). 


ABSTRACT REPRESENTATIONS 


All the tools and artifacts discussed in the preceding section have some iconic relation 
to the original situation in the flight simulator, and, therefore, are situation specific. In 
contrast, abstract representations are sufficiently removed from what they stand for so 
that they pertain to many situations observed (perhaps even every situation). This section 
focuses on two types of abstract representations that are common in the debriefing 
meetings: one type is oriented toward action and the other toward understanding in 
terms of human factors. 


Concrete Action Models 


In aviation, there exist many memory aids for dealing with a variety of situations. These 
come in various forms: some are more general whereas others are designed for specific 
situations. For example, the use of the phrase “Aviate, Navigate, Communicate” analyzed 
above is designed to orient pilots to prioritize flying and extraction in emergency and 
other unusual situations before considering navigation and communication (with air 
traffic control). Here, the final two syllables create a rhythm that facilitates recall 
of the mnemonic device. Similarly, “Pitch, Power, Performance”, “Power, Attitude, 
Configuration”, and “Pitch up, power up, Clean up” are designed to assist pilots in 
acting once a missed approach is declared, such as in the scenario featured in Chapter 
1 (“Missed Approach” box). Here, some mnemonic aids build (a) on alliteration, that is, 
on the recurrence of the first letter “P” or (b) on the recurrence of the preposition “up”. 
A number of acronyms in use concern decision-making, including GRADE (Gather 
information, Review information, Analyze, Decide, and Evaluate decision), DECIDE 
(Detect necessity of action, Estimate significance of action, Choose desirable outcome, 
identify actions required, Do required action, Evaluate effects), and DODAR, which 
stands for Diagnose, Options, Decide, Assign, and Review (e.g., FAA undated; Robson 
2013; Smith and Larrieu 2015). The latter is a general decision-making model used in 
a variety of domains. In fields other than aviation and other high-risk industries it does 
not tend to be crucial to act immediately. There is sufficient time to make a decision. 
Acknowledging that emergency situations in aircrafts tend to require immediate action, 
an additional component is sometimes added at the front end of the model. Thus, British 
Airways pilots (are taught to) use T-DODAR, which orients them to consider the time 
available before dealing with the emergency situation (Fortune et al. 2013). Other airlines 
precede the acronym by an F (Fly), which orients pilots to first act toward extracting 
the aircraft from the current situation before taking the time it takes to consider options, 
make a decision, and so on (e.g., Munro and Mavin 2012). 
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The (F-, T-) DODAR acronym is associated with different representations: simple 
list format, flow chart, or recurrent cycles of a spiral process. There are even situations 
where flight examiners consider the terms of the acronym as academic, not directly 
related to the world of piloting an aircraft, and provide pilots or trainees (pilots in 
type-rating) with equivalent lists that use the words they consider pilots to be more 
familiar with. The following scenario related to the F-DODAR acronym in the list 
format exemplifies the affordances of acronyms in general. 


Scenario 8.5 


FE: During this LOFT, we had an engine failure. Again, you acted in this situation 
according to our F-DODAR model. You flew the aircraft ((Points to “Fly” of 
the F-DODAR model, Figure 8.4)). You disengaged auto throttle, power up, and 
affected engine thrust idle ((moves finger down along the list of items under 
“Fly”)). You then called for the correct checklist. Doing so demonstrated good 
situation awareness. You started the checklist, doing the diagnosis. Then something 
happened: You missed an important part in the engine-failure checklist. What you 
missed was the auxiliary power unit. 


In this situation, the flight examiner accomplishes much of the orienting work 
toward the F-DODAR model. The flight examiner points out that the opening part 
was done according to the model, where the pilots dealt with the first item: Fly the 
aircraft. They then had oriented toward the second action: Diagnosis. As the subsequent 
description shows, it is here that something went wrong. The pilots had forgotten one 
entry from the checklist because they had stopped to make a pan-pan call and, when 
they returned to the checklist, had jumped ahead—which is what had happened in the 
Ansett New Zealand flight that subsequently collided with terrain (see box, this chapter). 

Now consider an alternate scenario where the initial part was not done to the flight 
examiner’s satisfaction. 


Fly the aircraft 


- Autopilot or hand fly? 
- Speed 

- Power management 
- Aircraft configuration 


Diagnose 
consider Options 
Decide 

Assign tasks 


Review situation 


Figure 8.4 The F-DODAR action model in list format. 
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Scenario 8.6 


The participants are watching a video clip recorded during their prior simulated 
flight. The flight examiner already has introduced the topic of the engine failure. 
Without further ado, the flight examiner orients the pilots to the debriefing tool, selects 
from the menu an episode that was previously marked for potential use during the 
training and examination session. The selected clip shows the simulator situation 
from immediately before the warning and stops it when the pilots begin with the 
checklist. The flight examiner points out that the pilots did not use the abnormal 
situation management model and states that in the simulator they talked about using 
the model and repeated the same exercise. The flight examiner asks the pilots to sit 
back and watch another clip, featuring the repeat. The flight examiner then stops 
the replay to orient the pilots to what they have seen, specifying that the clip has 
shown the implementation of the F-DODAR model (Figure 8.4). 

The pilot flying at the time places the index finger on the first item in the list 
under “Fly the aircraft”, “Autopilot or hand fly?” (Figure 8.4), while stating, “hand- 
flying the aircraft to 10,000 ft”. The index finger moves to the next item, “Speed”, 
which is followed, after a pensive pause, by the statement that this item actually had 
been missed. As the finger moves to the third item, “power management”, the PF says 
that auto-throttle was disengaged and affected engine thrust lever put to flight idle. 
They configured the aircraft and then moved to the checklist. The flight examiner 
acknowledges the account, and, while running the tip of the pen three times down 
over the four-item list, says: “So we got three out of the four”. 


This scenario exemplifies the production of an account of the past, which now is a 
denotable event. In its sequential unfolding, the account exhibits a structural mapping 
to the numbered list of action items that appear in the first item of the abnormal 
situation management model. Because the account is produced in reference to the 
event recorded and witnessed when the videotape of the debriefing tool is played, the 
operations performed in the cockpit are also and at the same time treated as exhibiting 
the same structure. Indeed, although the relevant part of the original event lasts only a 
few seconds, the highly structured procedures require the sequence to be flown in that 
way or else the flight examiner would conclude the performance to have been “messy”. 

In the case discussed here, the missing speed is the precise issue that is to make this 
fragment sufficiently stand out from the preceding flow so that it is easily remembered 
should the captain find himself in an abnormal situation again, not only in the simulator 
but especially while on everyday duty. The PF realizes that attention to the speed had 
been missing during the event; and it is only in the present that this omission becomes 
salient in conscious awareness. The item allows pilots to remember or recognize that 
they had missed something in the original event—not unlike what happened in Scenario 
8.3 when the pilots moved through the QRH checklist. In the case of the F-DODAR, 
the PF stepped through the items with the finger, verbally articulating what had been 
done and what had not been done. This in fact bears similarity with the use of Munsell 
color charts, where the users take an existing grid of hole-bearing pages, and seek a 
match between a sample and the color that surrounds each hole. The process therefore 
is similar to the way in which non-flight examiners assess pilot performances when 
they take neighboring performance descriptions from the assessment instrument to see 
which one provides the closest match to the data (see Chapter 3). However, there are also 
differences. In the case of the Munsell chart, the object compared to the referent (here 
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the F-DODAR model) is immediately present, whereas it has to be made present again 
in debriefing case. If the event already has been shown to pilots on videotape, the time 
between the replayed event and the comparison is brief. In the absence of a debriefing 
tool or when the pilots are asked to analyze what they have done prior to viewing the 
event, the work involved in producing the comparison becomes more difficult. Indeed, 
as shown above, the presence of the model itself supports the work of making present 
of the past. It constitutes, during the pilots’ self-analysis, a technology that shapes 
professional vision. The model functions as a norm. If the preceding behavior was 
appropriate, the norm is part of the because motive that explains why the aircraft and 
pilots have done well—i.e., because they have acted according to the model. Conversely, 
problems are explained to have occurred because the pilots have not acted according 
to the model, which, in this scenario, pertains to having missed to attend to the speed. 

The (F-, T-) DODAR model is also used projectively, suggesting what has to be done 
in such situations should these occur in the future. Flight examiners sometimes present 
the model and discuss it with pilots during the brief preceding the simulator session; 
sometimes they discuss it during the simulator session (in the recurrent training part); 
and they discuss it for the purpose of informing future performances that follow the 
recurrent training and assessment days. In all such situations, the (F-, T-) DODAR model 
is associated with the in-order-to motive (or plan). Here, the relevant action—which 
already has occurred in the retrospective situation and therefore exists as fact—is only 
projected: In future situations, the pilots also have to attend to the speed. The situated 
acting that follows orients itself according to an action in the future perfect tense as 
already completed and as having followed the (F-, T-) DODAR model. The unity of 
acting constitutes itself exclusively in this projection, and it therefore requires a sense 
of acting according to the model (Schiitz 1932). In those situations where the model 
is new to pilots, the model will be of help as much or as little as any other instruction 
read by someone who has never done what the instructions describe. Readers certainly 
are familiar with situations when trying to follow instructions (recipe), noting their 
inherent under-determination of what precisely has to be done to succeed. Perhaps 
unsurprisingly, flight examiners have been observed to ask for a repeat performance in 
the simulator after discussing why the first attempt at the task (1.e., to fly the emergency 
situation according to the F-DODAR model) was failing to meet desired levels. What 
pilots have just done, therefore, is now known to be inconsistent with the model, which 
provides a (stark) contrast to any subsequent performance that indeed will be said to 
have been consistent with it. 


Abstract Comprehension Models 


Inherent in and constitutive of the practices associated with the bi-annual recurrent 
training and assessment are the pertinent discourses. Particular assessment discourses 
and discourse elements can be observed whether or not airlines have explicit performance 
and assessment models. Flight examiners talk about decision-making, communication, 
situation awareness, and management with or without explicit performance models. 
Formal models may be verbal only or they may be presented in a diagram that then 
constitutes some perceptual order (e.g., Mavin et al. 2013). Such diagrams may not 
have a scientific foundation, especially when the relationships between concepts remain 
undefined (see Chapter 1). Here, the issue is not whether the models that airlines employ 
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actually make sense from a psychological, cognitive, or systems theoretic viewpoint; 
instead, the question is about use and function in concrete human practice. 

Consider the following scenario typifying high flight examiner to pilot talk ratios 
generally observed when some explicit performance model is used. In this way, pilots 
come to be confronted with model that they are not or little familiar with, and they 
have few opportunities to develop an associated discourse because almost always it 
is the flight examiner who articulates assessment using this discourse. Here, the flight 
examiner first produces a narrative of what was observed in the simulator. The first 
part of the account repeatedly articulates very good communication among the pilots. 
The flight examiner then refers the pilots to a graphical model lying on the desk facing 
the pilots (Figure 8.5), pointing with the pen to three words while articulating them: 
communication, decision-making, and situation awareness. Both the sentence structure 
and the movement of the pen are consistent with, and can be heard as implying a causal 
relation: Good communication leads to appropriate decision-making allowing situation 
awareness to remain high. 


Scenario 8.7? 


FE : Overall the LOFT was well done. You worked well together as a crew. Your 
communication was very good. There was a lot of discussion going on. From the 
right-side viewpoint it was good because the correct issue was articulated. You 
used the F-DODAR process: Fly the aircraft. Okay, what’s wrong? Let us have 
a look and diagnose what it is: Run the checklist. You discussed avoiding icing 
conditions. Let us keep climbing. You did this and extracted yourself from the 





Decision- 
Making 








ommunication 














situation. This was the end of the icing problem. You talked about going back: 
the weather in Milltown was not good, so we are not returning there. Newtown 
was suitable because an instrument landing system was available. You checked 

the weather. As a crew you discussed 

those core issues very well. So let us 

Awareness 

use the assessment model: Having 

[this communication], among the 

crew, then [your decision-making], 

took into account all the risks, and 

there was [situation awareness], MEA ae a rate e a 

throughout. f | i 

Typical in this scenario is the fact that there is no verbal exchange between flight 

examiner and pilots over and about the model. Such exchanges might have revealed 
the transactional relevance of the model—that is, the impact of the assessment and 
the model would have evidenced themselves in how the model is taken up in the 
pilots’ contributions to the exchange. Instead, pilots tended to listen to and observe the 
movements of flight examiners, as if these were articulating their assessment for the 
purposes of the records without consideration of the use that the model might have for 
the pilots’ work. In Chapter 10, there is a situation where the flight examiner asks the 
pilot to map an aspect of their performance to the human factors model, and the pilot 





> The grey-shaded and bracketed words coincide with the pointing gesture see in the image to the right. 
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flight 
examiner 


tool, 
artifact 


(Debriefing tool, 
F-DODAR, QRH) 





cockpit 


(Engine fire emergency 
situation) 


Figure 8.5 A typical model used in the cultural-historical literature emphasizing the “mediated” nature 
of action, where the role of texts (e.g., F-DODAR) between instructor/trainer, students, and the world 
represented in and by the text. 


fails doing so. As seen in Chapter 3, flight examiners themselves differ considerably 
in the ways they map observations onto the assessment discourse—a situation that is 
documentary evidence of poor decision-making for some is evidence of poor situation 
awareness for others. Moreover, even with explicit training and discussion of the use of 
such a model, rater agreement using this model does not increase (Mavin et al. 2013). 

What form of work might do the acts of pointing to the words? One important 
aspect is that there is further accentuation of the words themselves, which not only 
are heard but also can be seen. That is, pointing makes salient those terms from the 
discourse that are actually used to organize the assessment. Having the model in clear 
view may assist flight examiners—who, following airline policy, have to construct 
assessments in terms of the model—in articulating the assessment in all those terms that 
have to go on record together with supporting evidence especially in those instances 
that involve a fail rating. But whether the words and the discourse employing them 
makes any sense to the pilots cannot be known when debriefing and assessment default 
to the talking-head mode. 


Comparison of Action and Comprehension Models 


Different forms of models do different kinds of work as part of the debriefing situation, 
and they benefit participants in different way. Action models (e.g., [F-, T-] DODAR, 
DECIDE, or GRADE) not only serve to assess whether pilots have completed a sequence 
of actions embodied in the acronym, but also have a future, prospective orientation in 
that they orient pilots how to act in the case of an emergency. Comprehension models 
that articulate performance in the form of human factors may be less useful to pilots, 
especially when they have to act quickly rather than having lots of time to sit back 
and classify performance. Abstract models based on a set of theoretical concepts may 
serve the work of flight examiners, who therein have a set of concepts that can be 
employed in a principled manner (which, as seen in Chapter 3, may not be consistent 
across examiners and cases). 
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In the P-A-C and F-DODAR-related scenarios, explicit mappings of events onto 
action models are observed. Whereas the conceptual model explicates relations between 
human factors categories without specifying what should be done in any actual situation, 
action models projectively orient pilots toward dealing with the emergency. In the first 
of the two scenarios, it also serves as a referent against which actual performance is 
assessed. That is, pilots first are given the opportunity to hear or watch what they have 
done and then may be asked to “run through” this event in terms of the P-A-C, (T-, F-) 
DODAR model, or any other pertinent model in use (e.g., GRADE, DECIDE). The 
model therefore specifies and aids pilots to extract from the reconstituted experience— 
i.e., E, at time £, and as part of experience E (Figure 6.2)—those aspects that are 
consistent with the action model. In so doing, pilots are oriented toward the past in a way 
that allows them to determine in which way that segment from the simulator sessions is 
an experience from which they can learn. For example, pilots may become conscious 
(if they are not already) of the fact that they have missed noting speed, configuration, 
power management, and so on. That experience is a negative exemplar, an experience 
that should not repeat itself. It not only is an explicit aid in making aspects of an event 
present again, making these salient from what has been an open stream of impressions, 
but also is a tool to make present precisely those aspects relevant to articulate “that one 
property rather than another [that] was sufficiently dominant so that it characterizes the 
experience as a whole” (Dewey 2008). In the F-DODAR scenario, it was the missing 
speed that made the engine failure episode stand out. 

In the case of the action models, explicit connections are made between the list 
of actions (list of parts of a flow) and the list of items (flow chart) on paper. This 
relation is very different from that in the abstract understanding models, in the case of 
which flight examiners generally index events (e.g., “flight management system had 
to be reprogrammed”) and then note something like, “your knowledge was good” and 
“there was good situation awareness”. Unlike in the case of the action models, which 
provide descriptions of types of actions to be taken, human factors models abstractly 
classify performances into types associated with particular skills and knowledge. Flight 
examiners use these models as perceptual ground for their talk, indexically referencing 
them while talking about one of the categories named. Pilots never have been observed 
pointing to or gesturing with respect to these models. Their sole purpose is to make 
perceptually present the model and its components that come to be mobilized in the 
assessment discourse. In the instance analyzed here, three concepts from a model— 
communication, decision-making, and situation awareness—are indexed to the sequence 
of actions being recollected and connected by means of hand movements just as this 
has been reported in the literature (Mavin and Roth 2014a). 

The usefulness of action models goes beyond assessment situations. Training 
managers and flight examiners use such models literally to shape the future behavior 
of the pilots. Action models may be thought of as plans for actions. It is well known 
that plans do not determine future action in any causal sense (e.g., Schiitz 1932; 
Suchman 2007); and even the most highly trained and experienced scientists may not 
know what precisely they are doing when the “follow a plan” until some time later 
when the results of their actions are available and understood (Roth 2009). At best, 
action models, in the same way as other forms of plans, orient and disposition agents 
to act in certain ways. They project and orient to a field of action, providing agents 
with a sense contexture within which their actions are intelligible. Repeated cycles of 
training may be required until the situated actions come to be such that they can be 
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judged ex post facto as having been according to the plan. Initially pilots may require 
clues, such as a flashcard featuring a list of the terms with the letter of the acronym 
printed larger. Of course, having to look for the model and stepping through it takes 
away resources and time. Thus, when pilots are more advanced, they may go by rote 
through the mnemonic items: Fly the aircraft, Diagnose, etc. But even here, an additional 
cognitive step is required: naming the list item and then identifying what it requires 
(as this occurs when novices learn to drive a manually shifted car: push [clutch], shift 
[gear], release [clutch]). Thus, in the most advanced form, pilots act in ways such that 
after the fact they can be said to have followed the plan without actually having had to 
take recourse to the acronym. That is, speaking cognitively, they act in specific ways 
without doing the cognitive work of formulating these actions in consciousness. In some 
research traditions—enactivism and embodiment theories—pilots are then described 
as embodying the specific decision-making model. 


AFFORDANCES OF TOOLS IN DEBRIEFING 


Flight examiners tend to intuitively know that assessment itself is not sufficient to learn 
but that they have to work with the pilots and actually go through the flight simulator 
events to “pull out” any aspects that will help improve future performance in real flight 
situations. Together with the pilots, flight examiners talk through LOFT sessions or 
individual parts for the purpose of “maturing performance” for future situations. They— 
that is, those from airlines other than the one focused on assessment—often tell the pilots 
that doing the nitty-gritty thing, actually going through the preceding events, will achieve 
a different effect than simply being told that they have passed the examination. Such 
flight examiners inherently make a distinction articulated by pragmatic philosophers 
between the continuity of sensuous and inchoate lived-experiencing, which is associated 
with the life-constitutive transactions involving a subject and its environing conditions, 
and an experience, which is an already completed whole and therefore “carries with 
it its own individualizing quality and self-sufficiency” (Dewey 2008). The distinction 
is not only relevant to flight examiners, but also to a larger concern in research on 
learning and cognition about the role of the past in future performance in general 
and about the role that particular instructional practices and representations have in 
articulating between the two. This chapter provides answers to the question of how tools 
and artifacts support the effort to constitute an experience that is related to the pilots’ 
earlier inchoate lived-experiencing (much of which may have been forgotten or never 
been represented by pilots before). The purpose of debriefing therefore is to produce a 
memorable experience that informs and improves future practice. The two forms are not 
the same. This is so even in the case that for the pilots, some of the affective dimensions 
may be associated with some re-living of affects (such as affective flash backs). As a 
result of the joint nature of debriefing work, the constituted experience embodies an 
order of the type mimesis,, largely present in the narrative and the associated physical 
dimensions of the debriefing room. 


Do Tools “Mediate” Debriefing Meetings? 


The research literature is replete with studies that characterize events of the type 
observable in debriefing as having been “tool-mediated” or “sign-mediated”, making 
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reference to the use of the tools and artifacts as part of exchange relations. In so doing, 
these studies generally do not to provide accounts of the precise work that the concept 
of “mediation” is to accomplish for the analysis and how “mediation” functions. Studies 
suggest that the tool “mediates” the account provided about some past event; it might 
even be said to “mediate between the pilot and his memory” of the past event. This latter 
form of “mediation” may be attributed to the videotape, which then is considered to be 
a tool or sign in the sense that it stands between subjects and their memories, affording 
access to memory. This would be so because seeing themselves in the videotape provides 
a means to remember what precisely had happened, as experienced from a first-person 
perspective. Such studies may employ triangles of the type that have been used to express 
the relations between (a) people when signs/tools are introduced to an activity (where 
they may be said to “scaffold” the pilots into producing the account) and (b) people 
and their brains and memories (e.g., Vygotsky 1989). What such diagrams do not show, 
however, are the functional and genetic relations within the debriefing activity (Vygotsky 
1997b). Instead, implicit in such triangular diagrams is the comparison between two 
different forms of activity, one in which some sign/tool is present (represented at the 
apex of the triangle) and one in which it is not (see Roth and Jornet 2017a). 

Other types of studies suggest that the representational tools “mediate the 
discussion”, existing between the knowing teacher (here flight examiner), the learner 
(here pilot), and the world (cockpit). In this approach, three triangles are used 
including adult, child, text, and world to show mediation at work in learning to read, 
the third triangle mediating between (lying across) those two external to each other 
representing the child before and after instruction (Cole and Engestróm 1993). In the 
present context, the representation equivalent to this approach would include the pilot, 
cockpit (world), flight examiner, and F-DODAR (text) (Figure 8.5). In this approach, 
two mediational situations already exist (solid lines), one in which the flight examiner 
is said to mediate the pilots’ access to the real world (cockpit), the second in which the 
text (here representational tool and artifact such as F-DODAR) already mediates the 
flight examiner’s access to what has happened in the cockpit (world). The mediational 
relation intended by instructional setting concerns the pilots’ relation to their work in 
the cockpit mediated by the representational tool (e.g., F-DODAR), that is, the text that 
currently lies between them and the flight examiner on the table. 

Any two forms of activity—one with and one without some tool or artifact—are 
incomparable when unit (rather than e/ement analysis) is conducted. This is so because 
the presence of a tool changes the system as a whole (Figure 6.1). Saying, for example, 
that the F-DODAR or debriefing tool “mediates” the relation between pilots and flight 
examiners by determining, regulating, or modifying their forms of participation would 
amount to the same as saying that language mediates between the two parties. But 
language cannot mediate because it has a transactional function only if it is shared. 
There is an exchange only when the thing exchanged is a psychological reality for two 
(Vygotsky 1987). When the sounds or gestures of one person do not have repercussions 
in the actions of another person, they have no transactional relevance and fail to 
communicate anything. It cannot therefore be a third entity standing between the first 
two. Thus, it is more appropriate to theorize the tools and artifacts present in debriefing 
as integral parts of the activity. As parts they may not be theorized as independent 
elements between which some other entity can enter. As soon as we think about the 
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relation, we see that it consists of three parts or it does not exist at all: person A, person 
B, and the thing exchanged (words, signs). In the transactional approach, therefore, every 
part of the whole is part of every other part: there is no space or gap between any two 
parts into which some third, again independent part could step. Instead, the pertinent 
form of tools and artifacts arises from their integral nature to transactional exchanges. 

Action models specifically function in the same way as other plans, even when 
these are articulated in the verbal mode only: Plans function projectively, orienting and 
positioning pilots to act according to the situation without determining (causing) the 
precise forms and products of acting. Retrospectively, these same plans allow pilots 
to produce accounts for the purpose of evaluate the now factual action corresponds to 
the action described by the plan. With respect to future action, action models and flight 
artifacts (e.g., quick reference handbook, standard operating procedure manuals) have 
normative quality because they constitute a way of implementing constraints on what 
should and will happen—though there is no guarantee, of course, whether any future 
performance (qua situated action) will subsequently be deemed to have been consistent 
with the model. Indeed, after an initial training, where pilots have been exhorted to act 
according to a model, they may continue failing to do so. However, acronyms such 
as (T-, F-) DODAR, DECIDE, or GRADE are so explicit that they allow discussion 
even without the representation present. In the simulator some pilots explicitly refer 
to their airlines’ models to orient themselves as soon as a non-normal event occurs. In 
the debriefing session, the actual event can then be reviewed, retrospectively, as to its 
consistency with the model. 


Presentification of the Past 


Presentations and representations are important tools for the production of what will 
have been an experience. The questions addressed here are very different from those 
pursued in classical representation research, which tends to investigate how participants 
solve some form of researcher-posed problem using different diagrams—e.g., finding the 
date and time in different parts of the world using a time zone map or a circle diagram 
(Schnotz and Kiirschner 2008). There are two major functions of representational tools 
and artifacts in debriefing sessions. First, during the debriefing meetings, participants 
use different forms of representations (in the classical sense) with and over which they 
articulate verbal accounts of what happened during the preceding four-hour simulator 
session. These include, for example, videotapes showing the pilots at work, cockpit 
posters, action models, or a conceptual models of pilot performance. Second, aspects of 
the past are made present again by means of the movements that also have been present 
in the original event. This includes, for example, the quick reference manual, charts, 
or instruments (by proxy). Perhaps more importantly, while performing other parts of 
flying that were used (seen to be used) during the event, pilots and examiners move their 
hands, arms, and bodies in the same ways. In such situations, aspects of an event are 
lived-through again; and it may be this duplication that underlies true comprehension. 
Neuroscientists suggest “that one had to experience something within oneself to truly 
comprehend it” (Rizzolatti et al. 2006). Moreover, the same neuroscientific research 
shows that the recognition of a movement is based on having felt the movement before, 
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which is also what allows humans to feel the underlying intentions and emotions. Pilots 
clearly need to access (aspects of) an event again so they can learn from it. It does not 
suffice to narrate a flight simulator event unless pilots relate to it as their own. This is 
especially the case in situations where pilots use some tool or representation to articulate 
the relevant movement again. 

This chapter describes and theorizes the function of presentations and hand/arm 
and body movements as an integral dimension of the making present of a past event. 
In taking up their place in the kinetic melodies that constitute the ebb and flow of 
actions on the job, flight examiners and pilots place themselves again in the cockpit 
populated with material or virtual instruments and actuators. The special function of 
the presentations and representations derives from the fact that they are available and 
thus shape the constituted experience so that “we should think of what is experienced 
as what is available to a person, as what is available to a person from a place” (Noé 
2012). The debriefing meeting then serves as a second type of invitation, where 
aspects of a previous event play out again. This chapter repeatedly refers to instances 
where recollection of past simulator events occurs with the perception of an object 
that was part of the previous event. Presentations have a special role because current 
lived-experiencing is constituted by what is immediately given and present to which 
aspects of past lived-experiencing are admixed (Figure 6.2). When something is seen 
as a specific thing, perception has become an occasion for remembering. To be a point 
of discussion between pilots and examiners, the past event has to be present again for 
all participants, constituted as a joint object in the public space that they share. The 
presentations and representations foster the joint constitution of objects, in which the 
flight examiner assists pilots in making present again aspects of the past and doing so 
in ways that they are recognized to be the same. 

The role of objects in the presentification of the past has been captured in a French 
novel focusing on the remembrance of things past (Proust 1919). The novel is interesting 
because of its relevance to the social psychology of objectified experience. In the 
novel, the protagonist undergoes an instance of presentification when he takes into his 
mouth a spoon of tea in which a morsel of cake is dissolved. This lived-experiencing 
actually ignores and erases the distinction between perception and memory in the way 
suggested above. In the present context, the artifacts function in the way that the tea 
and the morsel of cake in Proust’s novel: they become integral to the project of joint 
remembering during debriefing of events pertinent for the future actions of the pilots. 
The representations used in the debriefing room are such objects that constitute an 
integral part of the joint project in which flight examiners and pilots engage together. 
Presentations, acting alone or in presence of the things from the cockpit, lead to a more 
active process of remembering than drawing on objects from verbal memory alone. 
Pilots know flying in the way most people know how to write, where the hand “feels 
the letter form in the very process of their production rather than as finished objects and 
remembers them as gestures, not as shapes” (Ingold 2011). Memory returns precisely 
when the pilots are in a situation where their parts in the overall kinetic melodies are 
triggered in the ways the writer’s hands are triggered to produce a set of words or sign 
in a single flowing movement. 

The notion of availability is central to more recent psychological and philosophical 
ideas about the presence of the past: the extent to which the past can be present is a 
matter of availability (Noé 2012). Such availability is a function of what is and what 
agents can do; in turn, what is and what agents can do depends on what is available 
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to them. From this perspective, then, the presence of representations (cockpit poster, 
abstract models, or [T-, F-] DODAR) and cockpit-related artifacts (charts, maps, or 
proxy instruments) in the debriefing meetings changes the extent and nature of the past 
that can become available in the present (1.e., as part of an experience). Crucially, the 
artifacts and representations change how (aspects of) the past events can be made present 
again for the purpose of reflection, assessment, and learning for the future. Making the 
past present again, in the context of aviation, means that the transactions of pilots and 
their world need to be present again. 

The cockpit artifacts and representations constitute physical presences. Much 
like a mnemonist may recall items when these are associated with images of walking 
through some part of the city or a landscape (Luria 1968), aspects of past events are 
themselves present again (rather than represented in some way). These objects are 
part of feeling the past in the present objectified in and by these artifacts. This occurs 
not merely in narrative terms, as in the examples that these authors provide, but in 
material terms identical to or proxy of the material aspects of the original experience. 
The physical aspects—gestures and movements identical to pilots’ skilled practical 
actions—constitute the bodily ground upon which commemoration occurs. The cockpit 
poster, too, is an artifact that allows the pilots to bring the past into the present by 
attending to the features of the cockpit as a place of action. 

The different tools and artifacts discussed here may be divided into different 
categories with different functions in the presentification of the past: conceptual, action, 
and circumstantial models. Conceptual models are removed from the actual practice of 
flying providing a means to categorize performance aspects—e.g., situation awareness, 
including the subcategories of perception, comprehension, and projection—and to 
explain actual performance by means of the connections between the categories. Thus, a 
flight examiner might explain that a lack of familiarity with the procedures (knowledge 
category) increased the pilot’s workload (management category) leading to a decrease 
in situation awareness. A conceptual model describes a situation at a considerably 
abstract level similar to the knowledge domains and clusters. Knowing that a particular 
item can be classified in the knowledge or communication domain of an assessment 
model does not help pilots solve a problem that they encounter during flight. On the 
other hand, an action model does suggest what to do next and how to organize their 
immediate responses. These models name types of actions that pilots have to do or 
series of objects that they have to attend to (as in power, attitude, configuration). After 
the fact, they constitute frameworks for evaluating whether the actual sequences of 
actions in the simulator corresponded to the models and where, if the case, problems 
arose for the purpose of becoming better (e.g., already on the following, second day of 
assessment) at performing such that the model accurately describes what pilots have 
done. The difference between a conceptual model and an action model is manifest 
in the use of nouns and verbs, respectively. These constitute different ways in which 
abstract models and action models relate to past and future performances, because the 
latter are tied to the movements of the pilots in relation to the material and social world 
surrounding them while doing their job. 

The third category of representations consists of artifacts that stand for aspects of 
the cockpit such that pilots may act out what they have or should have done. They have 
functions similar to the props in a theatrical play. This is clearly observed in the case 
of the cockpit poster, which serves as a ground with respect to which pilots and flight 
examiners orient and move as if they are in the actual cockpit. These representations 
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solicit movement sequences that themselves constitute a form of bodily memory. 
Similarities are apparent with the different ways in which simulation and paper-and- 
pencil testing situations relate to actual performance in grocery shopping (Lave 1988). 
Here, for shoppers and dieters, the simulations have a closer relationship and greater 
authenticity with actions during actual shopping experience than paper-and-pencil 
tasks, corresponding in the present situation to action and conceptual models of actual 
flying events. 


TOOLS AND ARTIFACTS IN THE ASSESSMENT FOR 
PROFESSIONAL DEVELOPMENT 


The purpose of the simulator sessions and the associated debriefing meetings includes 
the assessment of lived performances for learning, that is, for improving performance 
in future situations. Pilots are to learn from what they have done—in today’s high- 
fidelity full motion simulators, flying is almost identical to flying a real aircraft—for 
their future doings. The assessment in aviation differs from that in formal education, 
where students tend to be tested on what they are supposed to have learned in some 
course. In the simulator sessions, the pilots are subjected to flight situations that they 
rarely if ever encounter during their regular work. Pilots are not only tested but also 
expected to get ready for unanticipated non-normal events. However, the only times 
most pilots are exposed to and undergo non-normals are associated with the simulator. 
Despite the unusual nature of the task situations, poor performance will lead to negative 
consequences (e.g., being taken off line for retraining and in rare cases dismissal). Filled 
with incident after incident, the testing regime is so intense that the pilots emerge from 
the simulator with the sentiment that the preceding four hours have left but a blur. That 
is, although pilots might perform efficiently, proficiently coping with the demands of 
the situation, they often have not been aware of what has happened but is available 
to the flight examiners’ observations. Thus, if more than implicit learning is to occur, 
some of this inchoate lived-experiencing—evidently lived through by means of material 
(corporeal) practice—has to be made present again so that it can become an object of 
conscious awareness and reflection. 

The present chapter shows that there are two modes of experience. In one, debriefing 
meetings constitute situations in which pilots reflect on past events, where the past itself 
becomes an object of the present debriefing event and associated lived-experiencing. 
The order of the world apparent in these two modes also differs, denoted respectively 
as mimesis, and mimesis,. When the pilots do again during the debriefing meeting what 
they have done before, then the past is present but not in any reflective manner (i.e., about 
previous experience). Instead it is of the same kind that it has been in the simulator. In 
both situations pilots are confronted with, and perceive, some presentation invoking the 
same kind of forms of availability characterizing flying. But in doing again what was 
done before or what should have been done, the events in the debriefing ready pilots 
for future performances (see Chapter 9). 

In debriefing meetings, original artifacts from the cockpit and representations 
have a special role in the presentification of the past. Why might this be so? There are 
suggestions that “conceptual skills, along with sensorimotor skills and other kinds of 
understanding, belong to the means by which we accomplish perceptual contact with 
reality” (Noé 2012). If some of the artifacts from the cockpit are present in the debriefing 


Differential Effects of Tools and Artifacts in Debriefings 259 


room, requiring perceptual contact, this might then become the basis for producing the 
kinesthesis and senses that also operated during the preceding simulator flight. There 
is a form of corporeal memory, a concept that allows us to understand the shift from 
inchoate lived-experiencing to an experience. We may try remembering as hard as we 
want without success, until, as shown in the novel Remembrance of Things Past (Proust 
1919), a taste, smell, sound, or movement returns the past into the present. Corporeal 
memory and the replaying of events that occurs when pilots and examiners engage the 
movements and orientations of actual flying evoke the corporeal affective-volitional 
aspect characteristic of lived-experiencing. 

To be able to characterize something as learning from past practice for future 
practice requires investigators to show the presence of past performance in subsequent 
but changed performance. A translation has to occur from events of the order of mimesis, 
to future events of an order mimesis, linked together by an order of mimesis,. The 
purpose of debriefing practice is experiential learning. Just as the appreciation of art 
requires the recreation of the object on the part of the spectator, the appreciation of a past 
event requires its presence achieved in the process of presentification. In the experiential 
learning model, the description (observation) and reflection of past performance is 
followed by the formation of plans that lead to implications for new situations. However, 
this may over-intellectualize lived-experiencing, which never is intellectual or practical 
alone, but always intellectual, practical, and affective. Because “experiencing like 
breathing is a rhythm of intakings and outgivings” (Dewey 2008), the presentification 
of past events therefore means constituting a context in which the same rhythms of 
intaking and outgiving are solicited giving rise to the feeling of remembrance. There 
is therefore a close relation between past (flying), current (presentification of flying in 
debriefing), and future (flying) experience: practice, practice, practice. 

Chapter 5 shows that flight examiners continue to dominate in the accounting of 
previous experience—even though some pilots accompanied the examiners’ accounts 
by moving through (parts of) the action sequences currently narrated. But the present 
chapter provides the material for understanding why the pilots’ involvement in debriefing 
is of such importance. Much like the videotapes available in the debriefing tool, the 
acting out of kinetic melodies supports the presentification of past experience. Thus, 
debriefing meetings have to shift the emphasis from flight examiners to pilots, whose 
accounting contributes to producing the above-noted chain of repetitions. 

Participants in the aviation industry are interested in professional development 
and learning from experience. They are more concerned with the ways in which 
performance can lead to better performance than with theoretical models about the 
way the mind functions. Because lived-experiencing is closely related to availability, 
movement, and feeling rather than to representation, what matters to pilots is a change 
in the way it feels to fly in critical situations rather than knowing about the abstracted 
and abstract relations between different human factors dimensions of performance. 
In the context of the collaborative endeavors within which the current research was 
conducted, the findings of this study already influenced airlines with respect to how 
flight examiners are trained. Together with a consultant, flight examiners evolved a 
model that would allow pilots to be more involved in the recreation and analysis of 
salient events, facilitated by flight examiners who assist pilots in making present those 
aspects of previous performances that were identified as requiring improvement more 
urgently than other performances. 
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The fact that the flying needs to be understood in terms of a joint cognitive system 
presents a challenge to making events present again, because the actions of any part 
of the whole system depend on all other parts; there is a system flow that is not unlike 
the melodies of the different instruments of an orchestra. Making an event present 
again may require the presence of the other parts as well so that the past may become 
present again in the debriefing meeting. This is especially the case because what has 
been done or needs to be done is present in terms of the associated coordinated bodily 
movements. When pilots announced “My column”, the action of taking over initially 
may not have been present again during the debriefing session. But it may become 
present in the context of the representational tools and artifact and when flight examiners 
ask for the articulation of specific actions, which, in the kinetic melody of the cockpit 
as a whole, would then entail specific responses on the part of each pilot. Once the 
actions are present again (e.g., the column), the real problem can be articulated for 
the purpose of learning. Thus, a pilot who failed to provide sufficient pressure that 
would have produced a positive nose wheel contact as stated in the standard operating 
procedures may now move through the corresponding sequence, which might include 
pushing against some appropriate artifact (e.g., the desk) that resists in the way the 
control column would resist. 
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+ Information processing and constructivist theories of learning fall short because 
of the inherent gaps they introduce between body and mind, the individual and 
the social (culture), and knowing-that and knowing-how. The cultural-historical 
approach was set up to overcome all such dichotomies—though the false ways 
in which it is taken up often return scholars to the old ways of thinking about 


performance and cognition. 


e Inthe cultural-historical approach, anything distinctly human essentially exists in 
societal (social) relations. Thus, all higher psychological functions and personality 
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exist in the ensemble of the relations that produce society. Any higher psychological 
function at one point in the biography of an individual was a social (societal) 
relation with another person. 


+ The cultural-historical perspective emphasizes the role of teachers, because their 
relations with students may constitute the genetic origins of higher behaviors 
eventually observed later. That is, individual behavior has its origin in joint 
behavior. This phenomenon is referred to as sociogenesis. 


¢ In the instructional event, a cultural form of practice, which has been fused into 
one (instructor), is divided and unfolded into, and distributed over, two (instructor, 
instructed). 


+ Teaching (instruction) and learning are not two separate or independent events but 
form a single unit, here referred to as teaching | learning: If there is learning, then 
there is teaching; and if there is (real) teaching, then there is learning. Teaching 
and learning are two manifestations of the same event. 


e In teaching | learning, actions (which have conscious goals) are broken down 
into sequences of actions, each of which has a conscious goal. That is, what is a 
single melody of non-conscious operations for competent practitioners (e.g., for 
the instructor) is acted out as a sequence of conscious actions. The ultimate goal 
of the instruction, however, is to initiate the development of that single melody 
that unfolds as a whole once it has been triggered. 


n the context of the biannual recurrent training and assessment practices, debriefing 

generally is intended as an opportunity to review flight simulator events so that 

pilots may learn from any mistakes they have made so that they will not be repeated 
in the future—though, as seen in Chapter 5, the culture in one airline was such that it 
does not make maximum use of the debriefing session. There have been many studies 
of debriefing and its role in professional development. However, recent literature 
reviews reveal that there is a surprising sparseness of theoretical and empirical work 
on (a) teaching and learning processes and (b) the role of tools and technologies (e.g., 
Fanning and Gaba 2007; Tannenbaum and Cerasoli 2013). Yet in their mundane work, 
flight examiners often quite explicitly address the issue of learning at the end of a 
debriefing meeting, such as exemplified in the following vignette. 


Scenario 9.1 


01 FE : What are you taking away from this debriefing? I mean to say, was 
there something general that gave you an opportunity to learn from your 
mistakes? 

02 P : There may bea lot of little things. You know, four hours in the simulator is 
a lot of time. So may be I should read ahead of time the simulator session 
overview? 

03 FE : Okay, reading the overview. But would you consider doing the Civil 
Aviation Regulation 218 package? 

04 P : Notreally. Perhaps I might have a look at some charts. 

05 FE : Iwould strongly urge you to do the CAR-218 package. That will help you 
from a professional development perspective. 
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In this vignette, the general issue of the length of the simulator session is addressed 
(turn 02), which, together with the intensity of the exercises, mitigates what pilots 
remember and thereby what they can identify as a lesson that will improve their future 
performances. Here, the flight examiner’s turn treats the pilot’s reply as insufficient, 
by asking whether the preparation for the training and assessment should consider a 
review of the route qualifications of a pilot in command of a regular public transport 
aircraft as specified by the civil aviation authority (turn 03). After a negative reply, we 
can hear the flight examiner urge the pilot considering the route qualification package, 
because it would assist from the perspective of professional development. Whereas 
not all flight examiners are explicit to such an extent, the fact that they do tend to talk 
with pilots about events and provide suggestions about how to improve performance 
embodies the cultural object/motive of learning. 

Chapter 5 shows how in many debriefing cultures, the talking heads-mode still 
prevails. Flight examiners tend to talk a lot more than the pilots taken together. The 
learning theory implicit in this talking-heads practice is an information-processing 
paradigm associated with the idea of information transfer. In this approach, flight 
examiners give (emit) the information that the pilots are to receive and absorb. The 
latter are to store the information in their long-term memories and recall it from there 
in the future when required to apply it in some practical situation. In the field of the 
learning sciences, the metaphor of learning as information transfer and storage has been 
subject to critique from the late 1980s onward. As the information-processing paradigm 
waned, a constructivist (including various forms of situated cognition) paradigm took its 
place to become the new reigning ideology. Constructivism fundamentally assumes that 
individuals “construct” their own knowledge, conceptions, naive theories, conceptual 
frameworks, identities, and the likes (Duffy and Jonassen 1992). This generally occurs 
when persons make “meaning”, where “meaning” is treated as something non-physical 
(i.e., metaphysical) that exists in the minds of individuals. When it turned out that 
individuals would not be able to reconstruct on their own and from the bottom up all the 
knowledge that a society has produced in its cultural history, the learning sciences turned 
to social constructivism. In the social constructivist approach, people are said to learn 
by constructing something together in a dyad or in a larger group (interpsychologically) 
that the individuals then somehow transfer to the inside and construct for themselves 
(intrapsychologically). 

The constructivist approach in all of its forms has some fundamental flaws. First, 
because it theorizes knowledge in the form of mental constructs and metaphysical 
“meanings”, it inherently is based on a body—mind dualism. Among others, such 
ways of thinking about knowledge introduce into aviation a dualism that separates 
technical from non-technical skills, mechanical failure from human failure, and so 
on (Dekker 2015a). Constructivism cannot explain the role of the body in knowing, 
such as described in Chapter 1, because what the body does always is the result of an 
underlying idealization, a mental schema or conceptual framework that comes to be 
expressed through the body. How something non-physical can act on the physical body 
remains unexplained. Thus, it is not an appropriate theory for the phenomenon of kinetic 
melodies that consist of perceptual, performative, affective, and cognitive aspects and 
are coordinated across agents (e.g., Figure 1.7). Second, there is a dualism between 
the individual and the social (cultural). This is so even in the social constructivist 
(socio-cultural) approaches, where persons are said to be (a) externalizing, vetting, 
and negotiating their individual meanings in a group and then (b) internalizing the 
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group results. Here, the process of negotiation implies that individuals already have 
something constructed before (Radford and Roth 2011); and this something is the 
result of their own prior (subjective) construction. Again, it remains unexplained how 
something constructed internally could exist in the group that does not already exist 
for all individual members; and how does something immaterial (metaphysical) come 
to be shared within the public forum of a group? Finally, all forms of constructivism 
leave untheorized the fact that humans not only are the subjects of activity but also 
are subject and subjected to the conditions, which, in part, they produce themselves. 
It leaves completely untheorized that pilots may be carried away by the events, which 
may be seen in incidents such as the TransAsia flight GE 235, where the aircraft ends 
up in the river before the pilots succeeded in restarting the working engine that they 
had shut off (box, Chapter 10). 

There does exist a theoretical alternative in which both the body—mind and 
individual-social dichotomies are overcome—though poor translations of the original 
work have falsified and eliminated precisely those aspects from the original theory. 
The theoretical alternative exists in what generally is referred to as cultural-historical 
psychology, which subsequently developed in one of its versions into the cultural- 
historical activity theory now prevalent in the Western literature (see Chapter 1). In the 
cultural-historical approach, the main underpinning assumption is this: the essence of 
human Being is the ensemble of societal relations (Vygotsky 1989).! In other words, 
anything we ascribe to a person as an individual psychological factor has its genetic 
origin as human relation; that is, the factor appears in the form of a relation rather 
than as a thing in the relation. In contrast to the social constructivist and sociocultural 
approaches that still dominate Western psychological and educational discourses on 
learning (often referencing Vygotsky), therefore, cultural-historical approaches give 
primacy to the social. Central to the cultural-historical approaches is that social situations 
cannot be decomposed any further, for example, into self-sufficient individuals who 
interact and thereby produce the social. Instead, social situations are the smallest units 
that retain all the characteristics of society. Thus, it is illegitimate to investigate teaching 
(the actions of a teacher) apart from learning (the actions of the student), and learning 
apart from teaching (Holzkamp 1993).? The two happen simultaneously and constitute 
a relation between two people. As noted in Chapter 7, in the Russian language in which 
the cultural-historical approach to psychology was initially articulated, there is a term 
that refers to teaching and learning simultaneously: obucenie. Some educators, learning 
scientists, and educational psychologists import the term into English texts without 
change. Others use the combination of the corresponding English terms, writing instead 
“teaching-learning” or “teaching | learning” (Roth 201 5e). In this book, the latter form is 
chosen (see also Chapter 7). What happens in the debriefing room then can be theorized 
by one term, which, because it refers to the event as a whole, also captures that flight 
examiners learn and pilots teach during debriefing (or are part of the conditions that 
allow flight examiners to learn). 





' Societal relations are characteristic of society as a whole and are at the heart of generalization and 
scientific concepts; social relations are specific to smaller groups and the knowledge (practices) that 
they embody do not (have to) generalize (e.g., Il'enkov 2012). 

2 The fallacy is often apparent in lecture courses, when the instructor says something like, “In the last 
lecture we learned. . . ”. 
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CULTURAL-HISTORICAL ACTIVITY AND INDIVIDUAL 
DEVELOPMENT 


This chapter extends Chapter 7 in more fully articulating a cultural-historical position 
on the unity/identity of teaching and learning that occurs in debriefing meetings. This 
approach foregrounds the fact that higher psychological functions exist in the forms of 
social (societal) relations; and even if people write for themselves—e.g., flight examiners 
taking notes during the simulator sessions or pilots taking notes during debriefing—they 
are still oriented towards the social in the form of “‘the intimate other’, i.e., that which 
is ‘other’ within ourselves” (Mikhailov 2001). From a cultural-historical perspective, 
communication, because of its collective nature, is the lynchpin of teaching | learning 
events: “speaking is the central function—social band + psychological means. Cf. 
immediate and mediated relations between people. Hence digression: imitation and 
social division of functions as a mechanism of the modification and transformation 
of functions” (Vygotsky 1989). Here, imitation does not imply mindless copying (i.e., 
“aping” or “parroting”). Instead, imitation already implies knowing and understanding, a 
social psychological fact that has been confirmed in more recent neuroscientific studies 
on mirror neurons (e.g., Rizzolatti et al. 2006). The primacy of the social relation in a 
cultural-historical theory of the development of higher psychological functions finds its 
expression in the notion of teaching | learning, a single but double-sided process in the 
encounter of two people. The notion emphasizes teaching | learning as an irreducible 
unit, a single unfolding relational whole integrating and constituting the individuals as 
parts. This mutually constitutive process, teaching | learning, is brought about by the 
word, which, to play any role in the process, exists for (makes sense to) the institutionally 
designated instructor and learner simultaneously—or it is not a word at all (see below). 
The word is both social band and psychological means. The word, however, constitutes 
and represents the irreducible unity/identity of material (body) and thought; and it is 
the essential manifestation of the social nature of consciousness (Vygotsky 1987). As 
a phenomenon, therefore, the word constitutes the high road to overcoming the body— 
mind (psychophysical) and individual-social dichotomies. 

In cultural-historical activity theory, the work pilots do to get an aircraft from 
point A to point B is conceived of as part of one irreducible unit: productive activity 
(Chapter 1). That is, not only does the aircraft change its location but also everything 
else in the system and the system as a whole is changed, including the pilots. In this 
manner, the theory already includes the fact that on the job, pilots become better pilots. 
In the recurrent training and assessment-oriented (full-motion) simulator sessions, 
pilots get to fly under conditions that they do not normally encounter in their everyday 
work. These sessions therefore tend to be the only opportunities that pilots have for 
learning to fly non-normal situations. Here again, they learn to fly non-normal situations 
by flying non-normal situations. The order of their world constitutes mimesis,. In 
debriefing, the object/motive changes. Pilots’ normal but past activity (flying from A 
to B), now having become fact, becomes the object/motive (learning to fly). Pilots are 
to become aware of what has happened (to them) in the simulator and understand why 
events have unfolded in the way they have. In the process, their practical knowing is 
transformed, for it comes to include, among others, the flight examiners’ perspectives, 
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a third-person view and explanation. The descriptive and explanatory representations 
of work produced during the debriefing meeting constitute a new form of order, a 
new perspective: mimesis,. That work, theorized in Chapter 6 and exemplified in 
Chapter 7, is social through and through. 

The alternate takes on social psychology that historically developed from the work 
of Vygotsky may differ slightly, but they have a common core in how learning and 
development are understood. Persons are transformed in the course of participating in 
cultural-historically evolving society-specific human activities. They are transformed 
and become better at doing something even when they are not conscious thereof—such 
as when praxis improves praxis. Moreover, any psychological characteristic first is a 
relation with others, is lived as a relation with others before it becomes a relation for 
the self. When a psychological characteristic appears to emerge within the individual, 
then it always already is based on events that have occurred before (Schiitz 1932). The 
most decisive evidence for this theoretical approach comes from the study of deaf-blind 
children, who, before being tutored, do not exhibit intentions, do not intend to (learn to) 
walk on their feet, or do not reach out to learn by interacting with the surrounding world 
(e.g., Il'enkov 2007; Meshcheryakov 1979). Instead, everything they eventually know 
is the result of relations with others, where children not only come to feed themselves 
with spoons but also come to explore the spoon that feeds them. 

The social nature of human knowing becomes apparent when the normal ways 
of going about the current activity break down for one or another reason, people tend 
to exhibit precisely those forms of behavior that are characteristic of the origin of the 
higher psychological functions. Thus, for example, scientists—when struggling to figure 
what they are talking about and referring to while trying to make sense of the different 
representations that they have created—tesort to the same kind of fundamental process 
of creating mutual attention that can be observed when one-year-old infants participate 
with parents in reading events even before properly speaking a single recognizable word 
(Roth and Jornet 2017b). It may therefore be anticipated that in debriefing meetings, 
what already is one single form of behavior for instructors comes to be unfolded into 
the joint behavior with pilots. 

In cultural-historical activity theory, different levels are distinguished: activity, 
action, and operation (Leont’ev 1978). A fourth level may be conceived when the vital 
activity of the brain is included (Luria 1979). The first two levels—oriented to object/ 
motives and goals, respectively—occur on conscious planes. The other two levels occur 
on planes below conscious awareness. However, the movement that today consists of 
two actions in conscious awareness tomorrow may the result of a single action and 
subsequently completely disappear from consciousness when the movement has become 
an operation within a more-encompassing action. The example of shifting gears in a 
car with manual transmission may help. Initially, the entire process is decomposed for 
the learner. The intended shifting of a gear initiates identification of the present gear, 
followed by the pressing of the clutch, disengagement of the current gear, lessening 
of pressure on the gas pedal, moving the gear to the desired higher or lower position, 
followed by the release of the clutch and increased pressure on the gas pedal. For the 
learner, every step appears in consciousness, for even the release of the clutch may 
constitute a considerable challenge. Over time, the process turns into a single kinetic 
melody. Triggered by the in-order-to motive of shifting when the driver becomes 
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conscious of the need to do so, the shifting itself unfolds without requiring any conscious 
attention. More experienced drivers no longer think about shifting at all; indeed, they 
are unable to recall how many times or when they have shifted gears on a drive to the 
grocery store. Shifting happens in the same way as pedestrians walk through the city 
without having to be conscious about getting around obstacles (lamp posts, traffic sign 
posts, other pedestrians). The most experienced drivers shift without pushing the clutch, 
as this author has learned to do while driving trucks, which, complicating matters, 
normally also required double clutching. The example shows how initially complex 
sequences of actions initiated by conscious goals become unconscious operations on 
the drive to the grocery store. However, those unconscious processes may be brought 
back to consciousness again, for example, when parents or driving instructors teach 
others how to drive cars with stick shifts. At that time, the goal of pushing the clutch 
is articulated by the instructing individual but executed by the learner. That is, even 
a single action, the ensemble of goal and execution, comes to exist as social relation. 
Implicit in the notion of teaching | learning is the irreducibly joint nature of the 
work involved. A flight examiner alone cannot do debriefing. Individual pilots could 
reflect on previous experiences without others being present. But this is not an effective 
way of learning to debrief or of learning to learn from past performances. The joint 
work of debriefing is apparent from Chapter 7 designed to show how an assessment of 
various human factors requires the cooperation of examiners and pilots. But the joint 
work is perhaps most strikingly explicit when flight examiners and pilots concertedly 
perform actual (past) and more ideal (future) behavior. While accounting for the 
aircraft orientation in space during a flight event, flight examiners often use gestures 
symbolizing pitch and attitude, while pilots may simultaneously act out what they have 
done or should have done while flying the segment (Figure 9.1a). Here it is apparent 
that the two sets of symbolic movements are complementary, representing the same 
kind of first- and third-perspectives that were central to the account of making mind 
visible (Chapter 7). While narrating a flight segment, a flight examiner may use gestures 
exhibiting actions and gazing at instruments when checking that the bleeds are on; at 
the same time, concerned pilots act out the sequence in unison, same side, orientation, 
and position of the check (Figure 9.1b). Here, pilots and flight examiners are equally 
involved with body and mind, not only performing an event that simultaneously is a 
description of this event but also performing their practical understanding thereof. 
This joint work of performing events in the debriefing meeting (i.e., with an order 
mimesis,) is important for articulating what has happened in the past (i.e., where the 
order was mimesis,) and what should happen in the future (i.e., mimesis,). There is 
further professional learning and development possible, which is the focus of the present 
chapter. The episode derives from a type-rating training session. In type rating, pilots 
are trained to fly an aircraft initially new to them. After training and after having passed 
the examination, such pilots—who already have considerable numbers of hours on 
other aircraft types—then move directly from the training and their practical sessions 
in the flight simulator to fly as first officers on regular line duty. In the training session 
featured here, the pilot who was PF fails to properly execute the initial part of a go- 
around procedure. The flight examiner, here functioning as instructor, has marked in 
the notebook that the PF is not “nailing it quite yet” and has issues with the go-around 
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Bleeds are on? 





Okay, unusual attitudes, good. We 
actually come through the horizon 








Figure 9.1 Flight examiners and pilots frequently act out events in unison, taking complementary or 
the same perspective on the event. (a) First-(pilot) and third-person (flight examiner) views on a flight 
event involving abnormal pitch and attitude. (b) Pilot and flight examiner act out the same (first officer) 
perspective on checking bleed during an engine failure. 


routine, including the combination of rudder, trim, flight director, and power: “The PF 
was only going to the notch for the go-around procedure, but needs to go to the ramp. 
And before doing anything else, the PF looks down. The pilot should not look down 
because looking down makes the plane go down. The pilot needs to just feel the power 
lever hit something and then give it another handful”. In the instructor’s notes, the go- 
around procedure is highlighted for discussion during debriefing. In the assessment, 
although the PF has shown improvement, the performance was not yet at an acceptable 
standard level. The problem is particularly apparent whenever the aircraft is flying with 
one engine (after the second failed), but also is present when both engines are operating 
in normal condition. 





Teaching | Learning the Go-around Procedure 


A missed approach is called when pilots decide during the final stages of an approach 
to an airport that they cannot actually land because of some problem (e.g., cannot 
see the runway due to poor visibility). The associated go-around procedure includes 
numerous parts (Table 1.1), which are executed not unlike the different parts involved 
in manual gear shifting. Following the decision to do a missed approach, the pilot 
flying initiates a go-around procedure that is generally an integral part of an airline’s 
standard operating procedures and is also part of the manufacturer’s flight crew 
training manual. The go-around procedure frequently is taught in association with 
the memory aid “Power up, Pitch up, Clean up” (Chapter 8). But some go-around 
procedures may require not only more than the notch but, as the report concerning 
the crash of the Ansett Flight 703 Dash 8 discusses, even more power: “the most 
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effective response to a ‘hard’ warning from the GPWS requires the use of maximum 
power immediately rather than ‘go-around’ power” (TAIC 1995). The teaching | 
learning sequence described and theorized in Chapter 9 focuses on the first two parts 
of the go-around procedure involving the power lever and the electronic attitude 
director indicator. 

The top (pilot) and side views of the power lever provide insights in the relation 
between the different positions of the power lever (Figure 9.2). During a landing 
sequence, the power levers are back so that the aircraft does not exceed the required 
speeds. Thus, for example, at 1,500-ft altitude, the torque may have to be TO = 40%. 
On the final 100 ft to the ground, the power levers are further retarded to flight idle, 
and to ground idle once the aircraft is on the ground—to be further retarded in case 
the thrust is to be reversed. But in case of a missed approach, the power levers have 
to be advanced. As seen, they have to be advanced past the notch position forward 
right up to the ramp. In cases of “hard” signals from the ground proximity warning 
system (GPWS), even more power is often recommended (e.g., such as in the case 
of Ansett Flight A903, Chapter 8). This power is attained with the power levers in 
the most forward position (Figure 9.2). On the handle of the power lever, there also 
is a button that has to be pushed as part of the go around procedure. 

The second instrument that requires attention is the electronic attitude director 
indicator (Figure 9.3). The magenta-colored flight director command bars indicate to 
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Figure 9.2 Top and side view of the power lever (right-hand side of the aircraft). The setting of 
the power lever, together with the power management selection determines the actual torque. In the 
notch position, the power management may be set to TO (take off), MCT (maximum continuous), 
CLB (climb), and CRZ (cruise). 
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Figure 9.3 Principal elements of an electronic attitude and director indicator. The pilot flying 
aims to place the aircraft symbol onto the flight director bars by operating the steering column as 
required. In the position shown, the flight director requires the pilot to pull the steering column so 
that the aircraft pitches up to the 8° required by the go around procedure. 


the pilot flying where the aircraft symbol has to be for the flight to continue. When 
the go-around button is pushed, the flight director switches into go-around mode. This 
involves a “heading hold lateral mode” and “go-around vertical mode”. Concretely, 
this means that the horizontal line of the flight director jumps to that angle to which 
the aircraft has to be pitched. That is, when the line has jumped to 8°, then the PF 
pulls the steering column so that the aircraft symbol follows the command bars. 
When it does, this indicates that the aircraft now is climbing away from the ground. 
It is here that one of the problems in the Missed Approach scenario arises, where 
the PF does not push the go-around button (box, Chapter 1); it is also where there 
was a problem in the fatal accident of Lao Air Flight QV 301, where following the 
inappropriately programmed flight director bars led up to the disaster. When the 
go-around button is not pushed, the flight director bars continue to suggest heading 
downward even though the go-around procedure requires the aircraft to climb. 

The initial part of the procedure, according to the standard operating procedures 
(Table 1.1), requires a coordinated flow or joint kinetic melody from pilot flying 
and pilot monitoring (Figure 1.7). The flow includes the following aspects. 
Simultaneously with calling “Go around”, the PF (a) pushes the go-around button 
on the power lever (Figure 9.2), which changes the autopilot mode that operates the 
flight director (Figure 9.3) from “landing” to “climb” mode, (b) pulls the control 
column to rotate the aircraft to the required pitch attitude assisted by the flight 
director on the attitude indicator (Figure 9.3) by pulling on the steering column, and 
(c) pushes forward on both power levers from the flight idle across the notch to the 
ramp position (see Figure 9.2). Following this initiation of the procedure, there are 
many more parts of the sequence until the PF calls for an after-takeoff checklist, 
which then terminates the go-around procedure. 
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ENDOGENOUS PRODUCTION OF A TEACHING | LEARNING EVENT 


Fundamental to the cultural-historical approach is the understanding that humans, unlike 
animals, are not merely subject to their conditions but produce the very conditions to 
which they are subject and subjected. Human beings do not just fill prescribed slots 
(roles) in some social machinery, which then determines everything they do (i.e., they 
are not social dopes). Instead, as pointed out repeatedly in this book, humans produce 
their conditions from within these conditions, that is, they produce the conditions 
endogenously. This is also the origin of teaching | learning events. These require more 
than the actions from an institutionally designated teacher, who does something that 
miraculously leads to learning. For any teaching | learning to occur, joint work is 
required. There are many classrooms and other situations where teaching | learning 
does not occur even though an institutionally designated teacher is present and even 
though the participants are gathered in the institutionally sanctioned location for a 
specific activity to occur. In many inner-city (urban) classrooms, there frequently are 
so many different events that divert from any teaching or learning (e.g., Roth and Tobin 
2002). To use another example, a queue—in front of a bank counter or a supermarket 
checkout—is not a box into which people step. It is not a fixed “queue” schema that 
people act out, for they succeed even under the most mitigating conditions and when 
any semblance of a queue is missing. Moreover, while producing the reportable social 
fact they also provide everything required to get that work done successfully. In Vignette 
9.1, for example, the flight examiner, whereas accepting the pilot’s reply in saying, 
“Okay. Reading the overview”, also marks it as insufficient by commencing the next 
statement with a contrastive “but”. This contrastive connective leads into stating what 
is to be taken as the desired “take-away” message from the debriefing. That is, the two 
phrases do not merely have content but also organize the verbal exchange, here exhibiting 
that and how a reply falls short. Saying that a teaching | learning event is produced 
endogenously expresses the fact that participants in such social situations, debriefing 
meetings in the present context, not only talk but also produce the very conditions for 
the talk to unfold in the way it does. 

This section focuses on the endogenous production of a teaching | learning event 
that is part of debriefing. One of the goals to be accomplished in this debriefing meeting 
is the production of an account of the simulator event so that it becomes present again. 
That is, the debriefing meeting is concerned with the production of a first aspect of 
mimesis,. In the course of the meeting, this event then is to be critiqued and explained, 
thereby further working out an order at the level of mimesis,. This is frequently 
associated with opportunities to act out the required movements even though the pilots 
are not in the aircraft or simulator. Thus, as presented in Chapter 8, the pilots going 
over an approach chart or reading a checklist from a quick reference handbook are 
doing precisely the same that they would be doing in the cockpit: reading a checklist 
or going over an approach chart. 

In the exemplary event discussed here, the instructor initiates the episode by naming 
the event (“go-around”). After selecting the episode marked during the simulator session, 
the instructor starts replaying it for a while on the debriefing tool monitor describing the 
situation as a two-engine go-around and thereby distinguishing it from the single-engine 
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go-around that they have also practiced during the same session. The participants can 
see how while announcing the go-around, the PF moves the power lever forward to the 
first notch. The participants see a representation of the power lever during the flight on 
the right of the engine instruments (Figure 8.1). While the video shows the PF move 
the power lever, the instructor comments, “to the ramp there” and “pitch up”. It is only 
now that the power levers move from the notch to the ramp (Figure 9.2) and the pitch 
of the aircraft moves to the 8° mark (Figure 9.3). 

The instructor stops the video. But even before the instructor comes to a close 
in talking, the PF begins a statement about what needs to be done during a go-around 
and, in the second part of the statement, moves the hand in a way that simulates the 
movement of the power lever forward. But rather than stopping at the notch as it 
happened during the actual flight simulation, which is revealed in the video and on the 
representation of the power lever, the arm of the PF now moves to a fully stretched 
configuration. The instructor, while the arm/hand configuration moves forward for a 
bit (Figure 9.4a—b), articulates what the PF ought to be thinking about when “hitting 
something”. There is a pause, like the one that the participants have just seen in the 
video, and then a forward movement of the hand as if it were pushing the power lever 
while describing the action as “push past” (Figure 9.4b—c). Here, the description “push 
past” contrasts the stopping that can be observed in the video and the representation of 
the power lever, which remained for a while in the notch position. 

Here, therefore, a past event comes to be represented in verbal descriptions and 
in hand/arm movements. The particularity of these movements is that they are those 
required when actually moving the power lever forward. In the description, the same 
pause visible in the video is reproduced in the flight examiner’s symbolic movements. 
That is, the performance has an iconic relation with what the viewers are seeing on 
the monitor. The PF, however, already produces a different form of movement, that 
is, one that continues past the notch and somewhere that will correspond to the ramp. 

There is a first verbal description of how the power lever should be pushed 
associated with the hand movement, which is then repeated in subsequent exchanges. 
The instructor then formulates the upcoming instruction (“you got to go”), followed 
by a verbal description that instructs the PF how to move the power levers: first “into 
the notch” and then “another handful forward”. The “you got to go” is part of the work 
organizing the meeting, formulating (alerting) for the recipients that what will come is 
the description of the ideal performance to be achieved. This description simultaneously 
serves as an instruction for how to act. The actual description highlights that the power 
lever movement has to be more than just to the notch in the way that is seen on the 
debriefing tool. This latter part is formulated as a conscious goal: the movement is “not 
much it’s sort of just like another half handful forward”. This additional movement 
makes for about 11° in the angle of the power lever (Figure 9.2a). That is, the required 
movement is from flight idle position where the lever was (at 30°) through the 45° 
angle visible on the debriefing tool to the notch (69° power lever angle), and then from 
the notch to the required position in the go-around procedure, the ramp (80°). The 
movement is to occur in one single swoop, whereas the PF had the power lever sit in 
the notch position for almost six seconds. 
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(What ya need to think of (0.31) is you need te go (0.85)) 








hit something (0.48) 





Figure 9.4 Hand movements required for the “power up” required during a go around procedure. In 
the top panel, which reproduces Fragment 9.1, the instructor makes present what the pilot flying has 
done in the simulator and what can be seen on the debriefing tool (a—c). The pilot monitoring makes the 
same movement using the same hand/arm (d-e). The bottom panel reproduces the opening of Fragment 
9.3, where the pilot flying moves the hand/arm as required rather than in the preceding, incorrect way. 


The instructor then explains that it is crucial for the extraction of the aircraft 
from the situation with only one engine alive to operate at 100% power and therefore 
that the lever has to be moved to the ramp. The accident of TransAsia Flight GE235 
involved, among other issues, the incorrect movement of the power levers when one 
of the engines failed just after takeoff (see Chapter 10); and the accident investigators 
of the Ansett New Zealand flight that crashed near Palmerston North suggested that 
the PF should have moved the levers to maximum power rather than to the normal go- 
around power. Making sure that the power lever(s) are moved to the ramp or maximum 
power, and that the correct power lever is moved, therefore, is critical to the safety of 
the aircraft and its passengers. The instructor also points out that the failure is not a 
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major threat when both engines are alive before repeating the instruction once more, 
again providing a verbal description/instruction (“into the notch and a handful more”) 
and producing the hand/arm movement that takes the power lever from the flight idle 
into the required position at the ramp. 

The instructor notes that something else is to come. It is in addition to what has 
been said before and concerns the second aspect of the go-around procedure: pitching 
up the nose of the aircraft. As seen in the case of the Ansett Flight 703 accident, the 
fact that the nose of the aircraft was pitched up allowed the aircraft to slide over rather 
than run into a knoll. Of course, the idea during the go-around is to pitch up so that 
an accident does not happen in the first place. This is the central idea underlying the 
present teaching | learning sequence. The instructor says that the PF had gotten onto the 
flight trajectory in the simulator after being instructed to do so. At the same time, the 
instructor moves the bold-faced cursor in pointer configuration towards the horizontal 
part of the flight director (Figure 9.3) while saying: “pitch up again”. That is, the go- 
around is flown correctly when the aircraft symbol comes up to the flight director bar. 
In the written standard operating procedure, the initiation part of the procedure assigns 
to the PF the following actions: 


e Announce “Go around” 

e Simultaneously 
o Push the PL [power lever] GA [go around] button 
o Rotate to GA pitch attitude on the FD [flight director] 
o Set both PLs to the RAMP 

e Announce “Flaps 15” 


Thus, in the case of a two-engine go around, the call is followed by three items that 
have to be done simultaneously: pushing the go-around button, pitching up the nose to 
+8°, and pushing the lever to the ramp. The replay of the event on the debriefing tool 
shows, however, that the PF had stopped the power levers at the notch. At that point 
the instructor, using rhythmic intonation, uses a memory aid discussed in Chapter 8: 
Power up, Pitch up, Clean up. It is a memory aid in the sense that the words and the 
intonation are signs used to stimulate the brain and mind (Vygotsky 1989). The first 
two parts addressed in this mnemonic are precisely those that are discussed here in the 
debriefing meeting. The mnemonic is then reproduced twice. After an extended pause, 
this episode of the debriefing meeting is brought to an end by means of an assessment 
term (“so it’s cool”). 

As assessment and instructional sequence, this debriefing episode presented differs 
from most formal educational (school, university) settings in that the instructor not 
only provides verbal descriptions of actions but also produces, and thereby exhibits 
for instructional purposes, the relevant physical movements (Figure 9.4a—c). These 
movements, in their materiality, constitute something to be seen and to be imitated if 
required. The movements can thereby become a referent against which the actions of 
the PF are assessed; and these movements also constitute the ideal form that is to be 
instructed. What is to be learned is directly accessible to the learner, even though the 
PF does not perform the movements in an acceptable way for a pass rating here and 
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or over the next two days. The present performance is like a plan that functions as an 
in-order-to motive for the actions in the simulator; and that motive is not yet realized. 
That is, the actual, physical performance of the go-around is not yet such that the written 
procedure provides an adequate account of what the PF has done. This observation 
is quite in accordance with the research literature that shows that the simplest of 
instructions (recipes) may constitute a considerable challenge to the most educated and 
experienced professionals (Garfinkel 2002; Roth 2009; Suchman 2007). Even though 
the sequence may appear simple and even though the pilot has already seven years of 
experience as commercial pilot and has accumulated more than 3,000 flight hours on 
other aircraft, flying the procedure correctly does not come easy. 

In this debriefing fragment, the instructor not only provides a verbal description 
but also performs what the instruction names and describes. This is more than what the 
PF gets when reading the standard operating procedure: “Set both PLs to the RAMP”. 
Some readers might think that the instructions should be more explicit. But, however 
explicit an instruction always remains an in-order-to motive within open rather than 
closed future. There are many more ways in which instruction following fails than there 
are in which it succeeds. But being able to see the required movements together with 
the verbal description goes some ways: It is similar to watching a cook do what a recipe 
describes. In such situations, the intended learner has been afforded to actually perceive 
the relationship between verbal descriptions and the physical actions described. Given 
the considerable spatial accuracy of the flight-related movements that pilots display 
when assessing and instructing pilots (see Chapters 1 and 3), the instructor’s movements 
are available to the PF in the way they would have been while actually sitting in the 
cockpit. In this instance, the PF sat in the right-hand seat. The instructor’s movements 
are those of a pilot in that seat rather than the movements of a pilot sitting in the left- 
hand seat. The two pilots in training sit across the desk from the flight examiner (as in 
Figure 5.2c, and as apparent from Figure 9.4). Their movements, however, are not the 
mirror image of those that the instructor displays. Instead, both pilots are reproducing 
the movement with the left hand/arm; and this is the same orientation that the movement 
will be in the near future when both will be flying on regular line flights: in the right- 
hand seat and as first officers. 

In this episode, the instructor clearly orients towards the PF as intended recipient 
rather than to the other pilot, who was PM at the time. This is apparent from the body 
orientation that the instructor has with respect to the intended learner. However, as 
an instructional sequence, it is also available to the second pilot towards whom the 
instructor repeatedly orients by turning the upper body and head. Some observers of this 
part of the debriefing meeting tape suggested in the past that the PM was not involved. 
But the engagement of the pilot is indeed apparent throughout the debriefing meeting 
generally, as well as in an extensive verbal exchange and commentary in the episode 
immediately preceding the one presented here. Upon closer inspection, therefore, 
the claim of the PM’s non-involvement has to be rejected based on clear evidence of 
participation, such as in turn 14. 
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Fragment 9.1° 
14 FE: what=ya=need to think of (0.31) is (.) you need=te go 


(0.85) 
hit something (0.48) push past. 
(Figure 9.4a—b)| | (Figure 9.4b-c) 


PM: (Figure 9.4d-e 


Here, the hand movements of the PM produce—simultaneously in the way required 
by the standard operation procedures—the pull on the steering column with the right 
hand, which pitches the aircraft up, and the continuous movement of the left hand/arm 
combination, which moves the power levers to the ramp. The transcription shows that 
the final part of the movement occurs precisely when the instructor says, “push past”. 
That is, the movements of the PM are coordinated with those of the instructor, just as 
observed in many other debriefing meetings (e.g., Figure 9.1). The movements of the 
PM occur at about the same time when the other two participants also are moving their 
arms/hands, in a movement that here symbolizes the pushing of the power lever and 
pulling of the steering column. 


INSTRUCTING WHILE ASSESSING 


Any assessment requires the presence of the object to be assessed. In the present 
situation, the event to be assessed is made present in multiple ways. As seen in the 
preceding section, the participants in the debriefing meeting first watch a recorded 
performance, seen in the videotape and in the movements of the symbolic representations 
of the steering column and power levers (Figure 8.1). The instructor, too, reproduces 
the movements as part of producing a narrative description of what has happened. But 
the instructor also produces the hand/arm movements in the way that these should be 
performed such that they are consistent with the description that appears in the standard 
operating procedure. The PF, too, moves the hands and arms in the required way rather 
than in the way seen in the video. In fact, over the course of the 70-second debriefing 
episode, the PF produces the required movement four times. In contrast with what the 
PF had done in the simulator, the movements therefore exhibit an awareness of what 
should have been done and what should be done. The pilot became aware of something 
that previously was hidden while absorbedly coping with trying to fly a new type of 
aircraft. In the course of this episode, the participants therefore produce an account of 
what has happened, a critical assessment of what the PF had done while flying in the 
simulator, and an instruction of what the pilot has to do in the near future to successfully 
pass the examination that stands between the present instant and the future work as a 
regular line pilot. 





> The transcription conventions used in this chapter are found in Appendix B. The debriefing meeting 
fragments, originally presented in Roth, 2015e, have been adapted for the present purposes. 
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Assessing 


The fact that the instructor brings up the issue marks the go-around as salient, something 
worthy to be talked about. That is, not only the content of the communicative act but also 
its very production gives it a form of sense (Schiitz 1932). By naming the go-around, 
the phenomenon becomes the currently accented visible (e.g., El’konin 1994). It is to 
this accented visible that the participants now orient, and everything else in the setting 
and talk recedes to the background. The instructor completes the descriptive account 
provided so far with an assessment: Once the movement has been completed by “another 
half handful forward”, then it is “what it needs to be” (turn 19). 


Fragment 9.2 
16 FE: see you godda go | Tinto the notch 


PM: |(Figure 9.4d-e) 
FE: (Figure 9.4a-b) 
17 (0.34) Jand then you just needa (.) its not much its sort of jst like 
18 another] half handful forward 
(Figure 9.4b-c) 


19 and then its into what it needs to be. 


In this situation, the critique is articulated repeatedly. There are three normative 
statements—“you godda go”, “you just need”, and “then it’s into what it needs to be”. 
There are also repeated movements that are to be seen as demonstrating how the hand/ 
arm configuration has to be produced. Here it is apparent that the flight examiner does 
not explicitly describe what is wrong. The logic of the sequence of statements is such 
that the performance is correct (“what it needs to be”) when the pilot does what the 
normative statements describe: The movement of the power lever is “into the notch” 
and then “another handful forward”. As the events show, the PF actually articulates 
what is at stake (turn 24): The hands are there, while thinking about is happening, and 
that time is long (turn 24) before the PF had moved the power lever up to the ramp. 





Fragment 9.3 

21 PF: | mm sort of kind of that. 
(Figure 9.4f-g, sf 158) 

22 (0.75) 

23 FE: yfeaa:::. 

24 PF: m=hands there and i=m | thinking; what am I doing (0.15) oh yea; ramp. (0.47) yea 
am i at the ramp? (0.37) yea (0.18) and hhmy hharm (isnt?) so long. 


(as Figure 9.4f-g) 
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While verbally orienting the instructor to the hand movement (“sort of kind of 
that”), the PF reproduces the movement required to bring the power lever forward 
(turn 21). That is, the critique exists as the relation between what the instructor says 
and does, on the one hand, and what the instructed PF says and does, on the other 
hand. Because the instructor—instructed relation exists in and as talk, the critique is the 
relation. The critique, which is the awareness of the difference between what was done 
and what should have been done, in this instant, is the relation between two people 
(e.g., Vygotsky 1989). Readers should note that this statement is radically different 
from saying that the critique is articulated in the relation. In this latter approach, the 
critique somehow is floating in the group, whereas the point here is that the words that 
articulate the critique also constitute the relation itself. It is precisely here that the PF is 
afforded the possibility to grasp what is to be learned but which does not yet exist: flying 
in a manner so that the initiation of the go around as stated in the standard operating 
procedure actually describes what the PF has done. Because a plan or instruction 
(here the standard operating procedure) cannot ever be the cause of a situated action 
(Schiitz 1932), the question whether the PF knows how to fly the procedure can be 
assessed only the next time around in the flight simulator. From the cultural-historical 
perspective taken here, “it is impossible to know in general; it is possible only to know 
something in particular” (Il’enkov 2007). It makes no sense whatsoever to distinguish 
knowing something from its application. How is it possible to know something, such 
as a procedure, without being able to act accordingly? A pilot may know the text of the 
standard operating procedures by heart, which is something very different from knowing 
to act in a way consistent with the description that the standard operating procedures 
provide. Knowing the procedures by heart only means being able to repeat the words. 

The transcription in Fragment 9.3 articulates an insight in what needs to be done, 
moving the lever to the ramp. In this narrative, every action is depicted as if it were 
the consequence of deliberation. The pilot self-describes as sitting, wondering about 
the actions while the power lever rests where it is, and then that it should be at the 
ramp. We do not know whether this is precisely what has occurred at the time (1.e., 
Event E,); we take it to be a publicly constituted experience E; at the time ¢, of the 
debriefing meeting, i.e., as part of the lived-experiencing subsequently denoted as E, 
(see Figure 6.2). We therefore observe an awareness of the event as it has happened 
through the lens of what has been happening since reflected in the associated lived- 
experiencing. But, of course, the ultimate outcome is to be different. Proficient pilots 
fly the go-around procedure as one kinetic melody that is launched with the initiating 
procedure call. When replayed with the debriefing tool, the event shows that there has 
been a very long, six-second pause, and indeed the video reveals that instructor says 
during the flight simulator session that the levers have to go to the ramp and the nose of 
the aircraft has to be pitched up. That is, there has been a long pause between the two 
parts of a movement that should have been one. What the PF describes in the meeting 
as having sat there and as having thought about what to do may indeed have happened 
in this manner. When the instructor produces the movement required for pushing the 
power lever to the ramp, this can be seen as an actualization of what the procedures 
actually describe, and, in this way, may serve as a referent for the evaluation. 

Evaluation appears for a second time when the instructor notes that with two engines 
running—as it has been during the event in the flight simulator—the procedure as flown 
was “not such a biggie”. This is followed by an explanation: “it's a lot of power”. But 
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with a single engine running, the power lever must move forward to the ramp. The 
failure of moving the power levers to the ramp and simultaneously pulling on the steering 
column to pitch up the plan, because it now has been accented and has become visible, 
thereby also has become an issue. This is so because all three participants know that the 
PF has had this same problem before when the instructor failed one of the engines (i.e., by 
selecting from the simulator options an engine failure). Getting the procedure correctly 
flown is a real issue in single-engine go-around situations. In the case of the TransAsia 
Flight GE235, the pilot flying (rank of captain) had shown repeated weaknesses on 
the single-engine go around procedure, which the accident report uses as documentary 
evidence that the pilot’s “ability to handle an engine failure at takeoff was marginal” 
(Aviation Safety Council 2016), that is, a problem of the type that was at the origin 
of the deadly crash that ensued. Any airline manager wants to avoid having one of its 
aircraft in the situation of the TransAsia flight. The way in which the PF has flown in 
the simulator, here exhibited in the debriefing tool for all participants to see, therefore 
is problematic. In the examination that completes the type-rating training, it might 
well mean receiving a 2 (poor) rating on the procedure subcategory of the knowledge 
component of the airline’s assessment model (see Chapter 3). The associated verbal 
description in the assessment form establishes a correspondence between “Displayed 
lack of familiarity with procedure; correct procedure selected but executed slowly or 
with errors” and the associated score of “2” (poor). In the context of the debriefing 
meeting, the instructor has expressed the concern that the PF still does not satisfactorily 
perform the single-engine go-around procedure. 


Instructing 


In the episode, the instructor produces the full hand/arm movement required for the 
go-around, thereby contrasting the two movements seen in the videotape, of which the 
second included a verbal description (“to the ramp”). As featured in Chapter 3, flight 
examiners do not merely know how to talk about procedures but they act them out using 
many of the exact movements required for flying the sequence. The full movement 
produced in this debriefing meeting also serves as a model (plan, instruction) for future 
situations on line flights or in the simulator when this pilot is required to fly the go-around 
procedure. Similarly, the movements that the instructing flight examiner produces with 
the cursor to the flight director attitude indicator is an instruction (pointer) that tells 
the PF where to orient in future go-around situations to check whether the nose of the 
aircraft is appropriately pitched up. Each aspect of the procedure articulated in words 
and gestures serves as an instruction. Subsequent performances may then be compared to 
what has been said and shown. For example, later performances can be said to manifest 
learning when the normative descriptions provided here to orient in-order-to motives 
are recognized ex post facto to have been the because motives underlying what the pilot 
has done. If the future performance does not correspond to the normative descriptions, 
this may be explicitly articulated by one or the other participant—e.g., when the PF 
recognizes on the following day that the go-around still was not executed in the way 
that the instructor had taught it on the day before. Instruction here functions in both 
senses of the word: teaching and making known to a person what has to be done. Both 
these senses presuppose certain types of knowledge: that of (a) imparting knowledge 
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or skill (e.g., flying the go-around procedure to satisfaction) and (b) doing what the 
statements “to the ramp” and “pitch up” or the flight examiner’s hand/arm gestures 
describe (Figure 9.4a—c). 


A CULTURAL-HISTORICAL ACCOUNT OF TEACHING | LEARNING 


In this section two aspects of learning and development are developed. These two would 
be part of a program of a concrete human psychology concerned with how pilots know 
and learn while participating in typical activities such as debriefing that are organized 
from within. Learning, as assessment, are organized from within (endogenously), 
and made recognizable as such. That is, flight examiners coming across a videotaped 
debriefing meeting would recognize what is going on; and they can do so because the 
participants themselves make available to each other—and therefore to an observing 
flight examiners and researchers as well—everything required for pulling off the 
debriefing meeting in a recognizable manner. That is, debriefing constitutes joint and 
cooperative work: it is visible. Indeed, its visibility is a requirement for the joint and 
cooperative nature of the work. 


Social Relation 


In the cultural-historical approach, typical human forms of behavior, and therefore 
also typically human forms of cognition, are recognized to have been social relations 
before they became behaviors typical of knowledgeable individuals. The preceding 
sections provide glimpses from the learning trajectory of one pilot who, during the 
observational period, was not performing the go-around procedure in a way that it would 
be assessed as satisfactory during an examination. Now it could indeed be that there 
are pilots who somehow come to act consistent with the standard operating procedure 
without the kinds of relations observed in the debriefing meetings involving this PF. But 
how would such a pilot know that the procedure is flown correctly in the first place? 
The likelihood that any person by chance correctly produces the required movement 
sequences is close to nil. If (want-to-be) pilots do move through the sequence as required 
by the standard operating procedures, then this is so because they previously have seen 
it being performed. But as neuroscientists point out, to comprehend a specific movement 
in a particular manner requires the ability to move in this way (Rizzolatti et al. 2006). 
Sociologists add to this: we comprehend the world because it comprehends us, which 
it does (Bourdieu 2000), in the case of social facts, because of our contributions to 
the production of these facts. In any event, an assessment is required to ascertain that 
the procedure has been flown correctly. Those assessments themselves require social 
situations, such as those of formal pilot training and pilot assessment. 

In the present situation, the social practice of flying the aircraft-type-specific 
procedure, and deviations therefrom, exist as social relations. When the PF several days 
later flies the procedure in a manner that is sufficient for a 3 (satisfactory) rating, then 
this behavior has existed before as a relation with the instructor. The psychological name 
for this phenomenon is prolepsis, the representation of a future ability or development 
in the present (Cole 1996). During the exercise, the PF makes the call for “flap 15” 
prior to actually having moved the power lever to the ramp or rotated the aircraft to 
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the required pitch—although in both instances, some parts of the required movements 
had been done. In the flight simulator, the instructor then says “to the ramp there” 
and “pitch up”, which is followed by a beginning of the pitching up process and the 
movement of the power levers. In the following this event is considered while taking 
into account not only the actions—as done normally in transcriptions—but also what 
recipients’ perceptions required for replying. 

As seen in Scenario 1.3, the event of responding as a whole stretches from actively 
attending to the end of replying. When the flight simulator events are considered 
from this perspective, the PF begins to respond when actively attending to the flight 
examiner, who says “pitch up”. This situation is depicted, following the model presented 
in Scenario 1.3, in the following alternative transcription that takes active reception 
into account: 


Fragment 9.4 
FE (says): pitch up | Jesse 
06 |PF (hears): pitch up| Jesse 
PF (does): (pitch up) 
FE (sees): (Pilot pitching up) 


Thus, “pitch up Jesse” exists for both, and is integral part of responding, the second 
part of which is the movement that pitches up the aircraft. There is a command (plan) 
and an action. The two exist as relation between instructing flight examiner and the 
instructed PF (Jesse). As a result, a division of labor is achieved where what is fused 
into one for the instructor, an experienced pilot who flies the go-around well, now is 
divided and achieved in joint labor together with the PF. This “renewed division of what 
had been fused in one” (Vygotsky 1989) is precisely the cultural-historical position on 
teaching | learning events. What may occur now is that the PF, during the next flight 
simulator session when the go-around has to be flown, articulates the “pitch up” and 
then pitches up. But, as suggested above in the example of shifting gears, competently 
knowing to fly the procedure (mimesis,) means doing what is required without having 
to bring the order (plan, mimesis,) into the conscious mind. 

Fragment 9.4 also allows us to reconsider another aspect related to the social nature 
of higher psychological functions. In the cultural-historical approach, a word first is 
a command for others before it comes to have psychological functions as a means of 
persons for acting on themselves (Vygotsky 1989). Thus, “the relation of psychological 
functions is genetically [developmentally] linked to real relations between people: 
regulation of the word, verbalized behavior: power-submission” (original emphasis). 
The word, which is the ideal form of the action, and the associated practical material 
action are distributed across the relation. The relation between these ideal and the 
material forms is characteristic of, and reflects, a relation between real people. Later 
the relation comes to be attributed to individual things or people (e.g., Il'enkov 2012). 
Thus, when pilots proficiently fly this part of the procedure, then the genetic origin of 
the competence, which was a social relation first, has become invisible. We observe here 
why cultural-historical psychologists understand development as the individualization 


282 Cognition, Assessment and Debriefing in Aviation 


of social functions rather than in terms of the socialization of the (wild, raw, untamed, 
untutored, naive, or uncultured) individual. 

“Pitch up Jesse” simultaneously is in the mouth of the speaker (instructor) and in 
the ear of the recipient (PF). Responding, stretching from active reception to replying, 
therefore cannot be uniquely attributed to the PF. Pitching up exists not merely in the 
relation of instructor and pilot but essentially as their relation. In the same way, the 
movement of the power levers from the notch to the ramp exists as the instructor— 
instructed relation: Its intention exists in the words that order the pitch up, which is 
followed by the movement that pitches the aircraft upward. Five days later during the 
examination that marks the end of the type-rating training, an independent examiner 
observed the PF go through the required sequence that initiates the go-around procedure. 
At that point, what was a social relation in the event described here then existed in a 
way that can be ascribed to Jesse, that is, as one of the higher psychological functions 
exhibited by that person. We do not know whether the PF has been using during the 
intervening days the “Pitch up, Power up” formula to stimulate the action. If it had been 
in this way, then clearly the words of someone else, which initiated and described the 
actions, now would have been used by the PF to self-control, including the control of 
the brain activity required to get the procedure flown correctly. If flying the procedure 
correctly has not already happened without thinking about the steps, for example, by 
means of the “Power up, Pitch up, Clean up” formula, then it is likely to occur with 
continued practice and some time in the future. 

It is apparent from this analysis that teaching (instruction) and learning are not 
independent: learning is occurring when teaching occurs, and something deserves being 
named “teaching” only when learning actually happens. The expanded transcription 
(Fragment 9.4) makes this fact an accented visible. It may appear as if there is a simple 
cause and effect relationship between the instructor, who orders a pitch up, and the PF 
who is appropriately pulling on the steering column. As discussed, however, the relation 
between words describing an action and the practical action can only be assessed after the 
fact. Even most-experienced scientists have been reported as having intended following 
a certain scientific procedure and then, many hours later, having realized that they have 
not done what they thought they were doing (Roth 2009). Instead, the statement coming 
from the mouth of the instructor, to have any effect, has to be actively received. In that 
reception, transformations are occurring eventually leading the pilot to act in the face 
of the inherent ex post facto existing possibility to verify the correspondence. But this 
active reception will have been the first step in a learning process, which will become 
apparent as having occurred when the PF actually flies the procedure to the satisfaction 
of a flight examiner. 

There is more to this episode than this relation of an instructor and the instructed. 
In the debriefing meeting, the participants talk about and enact the relevant movements. 
In dissecting the whole movement, which exists for the instructor as one kinetic melody, 
a series of non-conscious operations now is made an object of conscious awareness: a 
series of goal-directed (conscious) action. The movement of the power lever is isolated 
from the whole movement; it is then further broken down into a conscious movement 
from flight idle to the notch followed by another “handful more” that takes it from the 
notch to the ramp. The debriefing meeting, the video clip, and the verbal and gestural 
production of the hand/arm coordinated movement allow something to come into 
conscious awareness for the purpose of making it subsequently disappear when compiled 
into an operation that will be a constitutive part of a much larger kinetic whole. This 
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whole, for seasoned pilots, is the entire go-around procedure that makes a single kinetic 
melody (see Chapter 1, Figure 1.7). It does not require any more conscious thought than 
driving the car to the local supermarket, an event to which a seasoned driver does not 
have to give a second thought. The debriefing tool, with its videotape and the visual 
representation of the power lever and steering column, assists in making present a past 
performance, allowing the development of conscious awareness of something that the 
PF has not been aware of before. 

So far the relationship considered existed in the flight simulator. But the debriefing 
meeting itself, which produced the representation of the previous performance to make 
it the object of reflection, also is a social relation. This social relation is established by 
and reflected in those very words that constitute the talk about and the instruction of 
the procedure. The two phenomena, the consciousness related to the procedure and the 
social relation making the meeting, are irremediably intertwined. The representation 
required for analyzing anything, a form of mimesis,, is the result of joint social work, 
in part existing in the jointly achieved unity/identity of {speaking | actively attending}, 
which alone can move the meeting ahead. The recorded performance becomes salient 
by the very fact of its selection, being named, and being played. This actively received 
solicitation and the PF’s subsequent actions together constitute the response. The 
consciousness related to the hand/arm movement, therefore, exist as social relation 
that makes the debriefing meeting before it exists as a psychological function that can 
be ascribed to the PF. 


Performance, Actions, Operations 


In Chapter 1 (Table 1.1), part of a go-around procedure is represented as a sequence 
of actions; it is in this way that the procedure is specified in the standard operating 
procedures. Chapter | also reviews the literature on the ways in which complex processes 
exist for pilots: as kinetic melodies that unfold once triggered, for example, by the 
command “Go around, set power, flap 15”. In the course of the analyses that happen 
in debriefing meetings, procedures are taken apart so that what normally are parts of 
an entire kinetic melody (i.e., operations) become separate conscious actions (as in a 
production system, see Chapter 1). This is so in the case of specific procedures, such 
as that required in a missed approach, or a more general decision-making procedure, 
such as the (T-, F-) DODAR process (see Chapter 8). The in-order-to motive is to make 
pilots aware of problems specific parts of the overall process so that the whole sequence 
is improved when one or more revised actions are compiled again into the melody so 
that it no longer requires conscious awareness. The following shows how a small part 
of the go-around procedure comes to be dissected into even smaller parts, followed 
by the articulation of a verbal device that is intended to allow the learning pilots to 
reconstitute the actions into the operations of a larger unit. 

In the cultural-historical approach, imitation and experimental unfolding of a 
single behavior and performance across a social relation are the main features in the 
production of psychological functions (Vygotsky 1989). Both aspects, imitation and 
social division, can be seen at work in this episode, here illustrated with the part where 
a whole movement is dissected for teaching | learning purposes. In this instructional 
sequence, what normally is one unit of coordinated movements—described in the 
standard operating procedures—comes to be unfolded into separate elements. Two of 
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the three parts in the “power up, pitch up, clean up” phrase are described, analyzed, and 
critiqued. First, the hand/arm movement required for pushing the power levers, which 
is to occur simultaneously according to the standard operating procedure, is addressed 
separately from the simultaneous movement required for pitching up the aircraft. This 
movement then is dissected further into two parts, the first taking the power lever from 
flight idle to the notch position, and the second continuing the movement to the ramp 
position. 


Fragment 9.5 


13 PF: °yea; need=ta=| do the go=around® 


(Figure 9.4f-g) 
14 FE: what=ya=need to think of] (0.31) is (.) you need=te go 
(0.85) 
hit something (0.48) push past. 


(Figure 9.4a—b)| | (Figure 9.4b-c) 


PM: (Figure 9.4d-e) 


The instructor’s movement appears in two parts, just as that of the PF seen in 
the video component of the debriefing tool. But if this two-part hand/arm movement 
had been what was to be the accented visible, then it would not have been other than 
a description of the original movement that is addressed as a non-example here. The 
instructor actually shows one extended movement, which takes the power lever from 
the flight idle position where it is, through the notch, and then right up to the ramp 
(Figure 9.2). This extended hand/arm movement has its verbal description in “you need 
to go, hit something push past”. That is, in the instructor’s version, the “Power levers 
ADVANCE?” is not a two-part hand/arm movement separated by a pause—represented 
in the transcription by the 0.48-second pause between the two parts of the verbal 
command (turn 14). The “push past” verbally describes what is to be the arm extension, 
thereby overriding any possible hearing and seeing of a movement consisting of two 
parts separated by some pause. The dissection is explicit in the difference between the 
movements that the PM produces, on the one hand, and those that the two remaining 
participants exhibit, on the other hand. In the former instance, the PM’s hand on the 
steering column also moves (turn 14, d-e), whereas it is at rest in the two other cases 
(turn 14, f-g, a-b). 

In this fragment, simultaneous with the verbal instruction, there is also a visual 
presentation of the corresponding hand/arm movement concurrent with part of the two 
parts of the verbal instruction. This presents an opportunity for a double strategy: the 
hand/arm movements, which in the cockpit do the actual work of moving the power 
lever and control column, here have symbolic function. As such, the hand/arm movement 
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affords imitation. The function of the speech in this instance is apophantic, that is, to 
make the relevant something show itself from itself so that someone else actually can 
find it (Heidegger 1977). The thing not only exists objectively but also reveals itself 
rather than being the result of an interpretation or construction.’ Speech marks this 
thing that heretofore has been a mere possibility of the perceptual contexture, allowing 
it to become visible. Indeed, it allows the phenomenon to show itself. What is to show 
itself is this: how a single movement can be thought as consisting of two parts even 
though it really is a single unit. Each of these two parts is associated with a physically 
sensible end point: into the notch and to the ramp. 

Second, the pitch up movement is addressed together with an orienting cursor 
movement to the flight director and attitude indicator where the aircraft pitch is 
represented (turn 39). That cursor movement, associated with the instructor’s changing 
gaze direction, accents and makes visible the instrument that is integral to the second 
part of the overall kinetic melody of the go around procedure. In bringing the flight 
director and attitude indicator into conscious awareness, what is to be one part of one 
conscious action, the go-around procedure, is articulated in terms of three (two plus 
one) actions: “to the notch” + “a handful more”, and “pitch up”. But the standard 
operating procedure requires the simultaneously executed pushing of the go-around 
button, movement of power levers to the go around position (at the ramp), and the 
pitching up of the aircraft nose (see above). If everything required had to be represented 
concurrently in short-term storage and operating space, all these different aspects would 
compete for attention and thereby overload working memory. It would require pilots to 
be aware of the notch, ramp, the goal of the hand/arm movement, the schema driving the 
movement and its actualization, the goal of pulling the steering column, the schema of 
the associated movement, the actualization of the movement, and the visual perception 
of the instrument (see the production system in Chapter 1). 

The instructor initiates this stretch of talk by placing the pointer to the electronic 
attitude director indicator (turn 39). In fact, there are multiple indirect references to 
the pitch before it is actually named and thereby constituted as the accented visible for 
their joint purposes at hand. Thus, the pointer moves to the instrument where the pilots 
may find the pitch of the aircraft at the instant shown in the videotape. Moreover, the 
instructor talks about getting onto the flight path, which implies the action of pitching 
the aircraft appropriately. 






Fragment 9.6 
39 FE: and=then (.) ye=know (.) just (0.28) see (0.57) 
you know 


you=re getting onto that flight [trajectory] after | 


said=t=you 
PF: (nods) 


+ A line appears when the eyes follow a physical feature (e.g., a black ink trace) while the focal point 
saccades on an off the physical feature. The line therefore is not a construction. The perceived line 
is the result of a movement sequence, which has followed the physical trace. 
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40 which is great but you just need to (0.81) you=know (0.26) jes a=bit like the 
old ways ye=know (0.89) power up pitch up 


(0.31) |clean up. 
41 PM: 2 hh hhhh* [| .h hh 
42 PF: “hmm 
(nods twice) 


In explaining the relationship between goal-oriented, conscious actions and non- 
conscious operations, the manual shifting of gears is presented above as a paradigmatic 
example. Something that is a conscious action with non-conscious parts for an 
intermediate driver, shifting the gear up or down, is decomposed for instructional 
purposes: Each operation is brought to consciousness and thereby turned into a conscious 
action. Through practice of executing the actions in sequence, the individual eventually 
comes to a point where the mere thought of having to shift gears serves as a trigger 
that initiates the entire gear-shifting movement sequence without ever requiring the 
presence in conscious awareness of any of its components. At that point, the procedure 
as a whole has become a kinetic melody that literally unfolds on its own once it is set 
off—i.e., unsupervised by a watchful consciousness. This debriefing meeting fragment 
shows precisely the type of instructional processes at work that lead to the compiling 
of action sequences into a single action where the former actions now have the status 
of operations. During instruction, the different (unconscious) operations are presented 
as three conscious actions. In the meeting, what had been two movements in the PF’s 
performance, two actions, are made to stand out with the ultimate goal of achieving their 
fusion into one continuous movement. For both issues, power and pitch, the instructor 
articulates multiple iterations of instructions for what to do. 


TEACHING | LEARNING AND THE DEVELOPMENT OF KINETIC 
MELODIES 


Functions of Instructional Sequences 


In formal learning contexts (e.g., schools), formative assessment tends to be 
distinguished from summative (final) assessment. In the context of debriefing, every 
assessment inherently has both functions. This is so because of the special nature of the 
culture, where the assessment situations rarely, if ever, are encountered in the course of 
everyday work. An engine flameout at takeoff is not a common occurrence. It therefore 
does not come as a complete surprise that the pilots of TransAsia Flight GE 235 did 
not get into the flow (kinetic melody). Indeed, the pilot flying in that incident had been 
required to redo an examination after having failed precisely this procedure; and he had 
difficulties appropriately acting in emergency situations (e.g., F-DODAR in Chapter 8). 
The simulator sessions and associated debriefing meetings both assess the pilots as to 
their current competency levels and prepare them to fly under non-normal conditions. 
This double function of assessment is apparent in the preceding analyses. 
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The playing, verbally describing, and analyzing of the videotaped simulator event 
has multiple functions. On a first level, it is a look at what the PF had actually done 
while flying this go-around procedure. But this performance is only a particular instance 
of the general performance of the procedure, especially critical when the PF would 
be flying the procedure with only one live engine, where the correct power lever has 
to be advanced. In the case of TransAsia Airways Flight GE235, the PF (captain) had 
shut off the live engine rather than moving the lever to the ramp (see Chapter 10). In 
the single-engine go-around, an additional complication has to be dealt with. The PF 
is not operating both power levers to the ramp but only the one that controls the live 
engine. That is, in the present instance, the details of the difference between the two 
procedures, go-around with one versus two engines, is not worked out—though the 
greater urgency of appropriately moving the power lever in the single-engine situation 
is named and therefore brought into the accented visible. Playing this part from the 
simulator session stands in for all situations in the simulator session where the individual 
has had to fly this procedure in the PF position. But making this performance the topic 
of the debriefing meeting is part of the instructional goal to initiate pilot learning. The 
repeated display of the full movement on the part of the instructor is a referent against 
which the preceding performance is judged; but it also is an instruction for future 
performances of this part of the go around. 

As instructional sequence, the debriefing fragment is more: it also includes the 
second pilot in training in the intended audience. That pilot responds in this sense when 
acknowledging the instructional comments and by correctly producing the required 
double movement. The second pilot not only acknowledges the instruction repeatedly, 
understanding it at a conceptual level, but also manifests correct praxis in the correctly 
executed hand/arm movements. 


Learning by (Immediate) Imitation and Social Division 


This episode illustrates what it means to know how to operate an aircraft. It integrates 
bodily knowing-how to fly an aircraft explicated and available for conscious awareness in 
knowing-that-type descriptions. The descriptions used are the same or similar to the ones 
that may be found in the standard operating procedures. Knowing-how exists for pilots 
in the form of flows, as kinetic melodies. Many a proficiently typing individual may be 
stymied when questioned about the location of a specific letter on a QWERTY keyboard, 
and yet without requiring any conscious thought will hit the key whenever required 
during the production ofa text. The typist’s knowing therefore exists in the typing rather 
than in the form ofa procedure stored somewhere in long-term memory. In the same way, 
aviation knowledge exists in the form of kinetic melodies. In the melody as a whole, 
performed by the ensemble, each member (i.e., pilot and the aircraft system) has an 
integral part—in the way an instrument has in a multi-instrument musical composition. 
Musicians would be unable to coordinate their playing if they had to consciously think 
about when they have a turn. In playing together, being attuned to one another’s play, 
there comes a point where a sense of the tempo of playing emerges as though it were 
external to the musicians (Sudnow 1979). That is, the announcement (call) “go-around” 
is the trigger for a whole event that is an integral and integrated response on the part of 
the cockpit as a whole (Chapter 1). If something does not happen—.e., if an operation 
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is missing such as pushing the go-around button—then the entire sequence may fall 
apart. Flight examiners then label the result to be an “untidy” or “messy” procedure. 

The kinetic melodies in a cockpit consist of bodily physical movements, perceptions 
(instrument readings, oral signals), and verbal “calls” that together get the work done. 
This work is collective, joint work involving contributions from all the parts of the joint 
cognitive system, much like a symphony requires the concerted, attuned actions on the 
part of all players. In the cockpit, the melodic flows become untidy when something is 
not said or done at the appropriate instant, incorrectly said, or missed altogether. There 
is a continuous flow where moving a switch, dial, or lever follows some sound-word or 
where gazing at a particular instrument leads to an announcement (e.g., the “positive 
climb” that follows after the PF has pitched the nose of the aircraft upward). If the 
pilots actually had to be conscious of following the procedure as spelled out in the 
document, it would be insufficient to produce the observed flow and synchrony when 
experienced pilots fly it. Just as can be found in musical performances (e.g., Weeks 
1990), the temporal context established by whatever has been going on before (which 
indeed constitutes the ongoing tempo), is a key resource for the synchrony observed 
in the coordination of the cockpit as a whole. Type-rating training intends to prepare 
pilots new to an aircraft type to become players in the orchestrated performances 
of such collective melodies. The purpose of the instruction in this specific episode 
presented here is to assist the two pilots-in-training, who are new to the airline and the 
aircraft, to compile currently conscious actions into larger, single units that constitute 
the response to difficult situations. The purpose of instruction—including flying in the 
flight simulator and debriefing what has happened—is to evolve patterns that allow 
these jointly produced, synchrony-requiring kinetic melodies to arise. That synchrony 
is produced neither from within the individual pilot (who follows) nor from without 
(by some agent making the pilot to act); instead, that synchrony is precisely the result 
of the joint production of inner and outer forces. 

The present fragments from one debriefing meeting manifest the ways in which a 
social relation including examiner/instructor and pilot is the form in which a complex 
behavior first exists. The social relation is the first appearance of the integration of the 
simultaneous movement in the life of the PF. Participating in the social relation sets 
up the PF to get the feel of the whole behavior even though it is still inaccessible on 
the part of the individual. In the cultural-historical approach taken here, the second 
important aspect of teaching | learning consists in the opportunities to learn by means 
of imitation, an active process requiring understanding (recognition) of what is depicted 
in a hand/arm movement rather than a passive process of blind copying. As suggested 
above, the cognition of an action requires the ability to execute this action; indeed, 
the mirror neurons involved in perceiving some action or object are the result of the 
agent’s own previous actions (Rizzolatti et al. 2006). In this episode, the instructor 
offers an opportunity for imitation by using the left hand and arm in the movement, 
which is what the right-seated pilot would have done. The instructor normally would 
have used the other hand and arm had the PF been habitually sitting in the left-hand 
seat (Chapters | and 3). 
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Functions of the Debriefing Tool in Teaching | Learning 


In this meeting, the debriefing tool is a resource offering multiple ways to make present 
an event to be described, analyzed, and critiqued. It is a tool in the production of order 
at the level of mimesis,. It affords the participants to view and hear themselves, as 
recorded by the camera mounted in the simulator. The video permits seeing the grosser 
hand/arm movements, though finer details including the actual positions of the power 
lever are not clearly visible in the video. This is so at least for the particular mode in 
which the video has been used; it can be improved when the window for the video is 
enlarged to the size of the monitor (the video itselfis very grainy). The instructor points 
to the attitude indicator rather than making reference to any action or to the steering 
column represented. In this way, the pilots are invited to find for themselves what 
there is to see. What can be seen stands for itself and does not have to be represented. 
Pointing to the aircraft symbol, which can be seen as aligned with the flight director, 
has good reasons: neither the video nor the steering column representation shows the 
minute movements by means of which the aircraft nose was pitched up. However, the 
movement of the aircraft symbol from the horizon (i.e., as seen in Figure 9.3) to the 8° 
position of the light director is clearly visible in the representation of the instrument on 
the monitor (1.e., as seen in turn 39). Particularly important is the fact that the electronic 
attitude director indicator (EADJ), though evidently a representation of the actual 
instrument, is different from a mere representation. This is due to its high fidelity and 
iconicity: pilots see in the same mode what previously has been perceptually available 
to them while flying the aircraft. Although the mode is the same, however, there are no 
guarantees that the awareness of the EADI in the simulator (W"") and the awareness 
of the EADI instrument in the debriefing room (Void ) are the same. Nevertheless, in 
both instances awareness of the instrument is involved rather than the awareness of a 
representation thereof (e.g., Husserl 1928). The debriefing tool exploits the sameness of 
the form of awareness, that is, it offers an opportunity to view an aspect of the flight in 
the same mode that was available to the pilots during the past flight and will be available 
during future flights. The EADI appears in the lived-experience on the horizontal axis 
extending from the past to the future rather than having to be made present by some 
form of recall (Figure 6.2). It is in this manner that the teaching | learning opportunity 
is like that arising from the advancement of the hand/arm during the debriefing meeting 
in relation to this movement during a flying situation. The representation of the power 
lever on the debriefing tool (Figure 8.2) is cognitively different from the electronic 
attitude director indicator because it solely appears in the form of an iconic representation 
without affording the equivalent identical sensory (haptic) modalities (pushing, feeling 
resistance) as the EADI. By means of the debriefing tool, some aspects are as if self- 
present in the meeting, and, therefore, do not require re-presentation. They can be 
referred to when and as needed. 

In this debriefing episode, the participants orient to the debriefing tool, which, as 
a whole or in its parts, becomes the object of perception. The participants then orient 
towards and address each other. While doing so, the debriefing tool still displays the 
image from simulator flight at the instant where it has been stopped: The PF with the 
left arm stretched forward, the power lever representation in the ramp position, the 
aircraft symbol on the flight director line, and the aircraft seen from behind heading 
in an upward direction (i.e., nose up). It is in this configuration that the debriefing tool 
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serves as the ground against which the debriefing talk stands out (figure). The image 
synecdochically stands for the past event as a whole. The adjective “synecdochical” 
means that a part of a phenomenon stands for the whole phenomenon—such as when 
the graphs research scientists produce stand for the entire research project, methods, 
data sources, and so on (Roth 2014). The synecdochical nature can be seen in the 
pointing gesture that makes the electronic attitude director indicator part of the accented 
visible. The issue to be discussed is the delayed process of getting from 0° to 8° pitch; 
in addition, the power lever needs to be discussed at the same time because its full 
movement also was delayed. The images of the indicator and power lever stand for the 
entire issue critiqued, the delay and prompt-requiring action that brought the aircraft 
onto its proper flight path. 

In its entirety, the episode exemplifies one of the advantages of the debriefing tool 
over other means of making an event present again—e.g., the verbal mode alone. When 
the videotape plays and the instruments display what was made available during the 
flight event, everything is literally present again in the debriefing situation. This recorded 
account of the original event literally stands for itself: It synecdochically is present in 
the image of the flight at the point that the replay is stopped. Participants may obtain 
certain information simply by looking at the appropriate place in the display (see also 
Chapter 10); for example, they can verify that the aircraft indeed has the nose pitched up 
to 8°. Verbal, gestural, or cursor referencing may indicate any aspect, thereby bringing 
it into the accented visible. 

In Chapter 5, the point is made that pilots tend to emerge from the simulator 
sessions drained, having forgotten much of what has happened, what they have said, 
and what the flying conditions and instrument reading had been for any given event. In 
this situation, the debriefing tool provides multiple re/presentations for a particular flight 
event. This is made present again in this way, unhampered by possible interferences that 
often plague personal accounts. Even the most seasoned pilots—with 20 or 30 years of 
experience and having become flight examiners—can be observed recounting events in 
ways that contradict the evidence available from the debriefing tool: They can be found 
saying to have done something which the videotape clearly shows that they have not 
done—such as completing a part of the standard operating procedure (see Chapter 10). 
There are irremediable states of affair that are the same whenever the event is replayed 
on the debriefing tool, whether hours, days, or months have passed since it originally 
occurred. (Actually, the simulator records in the participant airlines were deleted because 
of contractual arrangements with the pilot unions; deletion makes it impossible for 
courts to subpoena the records in case a pilot is involved in an accident.) In the meeting, 
the debriefing tool allows flight examiners and pilots to be held accountable to what 
appears on the monitor. Whatever appears is taken as an indisputable matter of fact. 
Thus, flight examiners revise assessments when they can see, or are challenged about, 
the precise nature of simulator events (see Chapter 10). The debriefing tool is a form 
that facilitates memory: initially as an artifact placed between the person and the thing 
(the simulator event), then between the person and his/her memory (Vygotsky 1989). 
As an external memory device, the debriefing tool is virtually incorruptible. 

An event made present again in indisputable form can be described, contrasted with 
descriptions of what ought to have happened, analyzed, and assessed. In this way, there 
are considerable affordances to the production of mimesis,, narratives that integrate 
practical understanding and reflective explanation of previous performances. These 
narratives are resources for the subsequent improvement of praxis, which occurs when 
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theory (mimesis,) is brought back into and thereby transforming praxis (mimesis,)— 
though a considerable distance remains to be covered from comprehending what has 
to be done to actually acting in the corresponding manner. In the case of the PF in this 
chapter, it took several more days before the actual go-around performance was such 
that the standard operating procedure provided an appropriate description thereof. This 
match did not exist on the training days prior to the final examination. When the event is 
indisputably present, it may serve as the ground over and against which the debriefing 
talk occurs. There is a mutual constitution between what appears on the monitor and 
the actions currently occurring in the debriefing room. There is indeed a lot present 
on the monitor of the debriefing tool, but it is the actual talk that brings one or another 
thing into the currently accented visible. 

There is a tendency in the research literature to theorize some objects, such as 
the debriefing tool, as mediators of social interaction (see Chapter 8). In the present 
instance, it is quite apparent that anything available on the surface of the debriefing 
tool monitor is not “mediating” between the instructor and the pilot flying. If the 
debriefing tool were a mediator, then it would be external to both participants. Instead, 
as illustrated in Fragment 9.4, anything that has transactional relevance is indeed 
common to the participants: The relation involves both participants and the tool as 
integral parts. In the logic underlying cultural-historical approaches, any perceptual 
and conceptual differences that may appear in the course of the exchange are due to the 
things displayed (e.g., Il’ enkov 1977). Just as a commodity in economic exchanges, the 
word exchanged may be one thing for a first and another thing for a second participant 
in the verbal exchange. Important here is that in the exchange, the thing is such that it 
can indeed manifest itself in different ways—which is the very reason why it may be 
different for the participants. Different understandings of the same word are like the 
different appearances of the same object in the two eyes—conditions for the appearance 
of consciousness and depth perception, respectively (Bateson 1979). It is precisely 
because anything that appears on the display is common to the interlocutors that it may 
serve for instructional purposes, that is, for the purposes of “handing down” cultural 
practices. The instructor’s presence affords the jointly produced actions to take on a 
well-known and recognizable shape; and it affords corrections whenever the shape 
of the actions falls outside of the range of accepted practices. In this way, the social 
relation between the instructor (flight examiner) and instructed pilots is the origin of the 
behavior subsequently observed when the pilots have graduated and have commenced 
their line duties. 
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KEY POINTS 


e Situation awareness is a widely used concept for describing the performance 
of agents in high-risk environments where being aware of, understanding, and 
projecting from the current context is of utmost importance to risk management. 


e Situation awareness is a contested concept. There are suggestions that it should be 
considered as something characteristic of (joint cognitive) systems rather than of 
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individuals. There are also suggestions that situation awareness as such does not 
exist, leads to blame games, and is theoretically fruitless. 


e Situation awareness may be understood simply as a shorthand feature that 
marks certain events for closer investigation. Even though it may not exist as a 
natural phenomenon, situation awareness may be used as an index for looking at, 
understanding, explaining, and teaching certain behaviors important to flight safety. 


+ There are special cognitive (learning) opportunities when instruction moves from 
telling to performing events generally and performing the content of instructions 
specifically (e.g., as stated in the standard operating procedures) to be emulated 
on the part of recipients. 


+ Multimedia, such as the debriefing tool, are not mere representational devices but 
have important functions, which include their (a) unchanging account of what has 
happened, (b) transportability across contexts (1.e., from simulator to debriefing 
room), (c) nature as backdrop and context for producing the debriefing meeting, 
and (d) nature as content to be included in the narrative description, explanation, 
and evaluation of past performance and instruction to inform future performance. 


ituation awareness is a concept that one can find in many places: in academic 

discourses, in accident reports, and in the talk of flight examiners conducting the 

bi-annual recurrent training and assessment sessions to which commercial pilots 
are subject throughout their flying career. The fact that “everyone” appears to be using 
this concept as an integral of part of everyday work, however, does not guarantee its 
scientific validity, as pointed out in the editorial introduction to a recent special issue of 
Cognition, Technology, and Work (Carsten and Vanderhaegen 2015). In this introduction, 
a number of legitimate questions are posed concerning the product or process nature 
of the concept, its reliable measurability, and its relationship to other concepts used in 
the social and medical sciences (e.g., to mindfulness) or information and organizational 
sciences (e.g., to sense-making). 

Aviation pilots themselves may not or not frequently employ the term but instead 
talk about concrete situations where they have not been aware and understood something. 
Consider the following statement, which concerns the realizations of a seasoned captain 
that have occurred during a debriefing meeting: “I was the flying pilot . . . where the 
normal up trim and autofeather did not work. I knew it hadn’t autofeathered; but I had 
no idea that the engines had not spooled up properly. I never realized until we were 
shown on that debriefing tool. . . . So it just highlights to me how, as the crew, we are 
not going to be able pick up all the events”. Here, the pilot talks about coming to know 
something that was unknown during the flight itself. Part of the failure was known, 
the failure to autofeather, but the other part, the failure of the system to up trim the 
operating engine, has remained unnoticed. The pilot concludes: “the crew are not going 
to pick up all the events”. Not picking up, not perceiving essential aspects of a flight 
situation, is an integral part of the way in which scholars define situation awareness 
generally, and its perceptual component more specifically. This is one of the reasons 
why the concept has so much currency in the industry. 

The concept is not without controversy. Given that it often appears in accident 
reports where human operators generally and pilots in the present context specifically 
are blamed, its legal and political dimensions also need to be considered. Indeed, its 
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use may hide much deeper system failures that are then pinned on humans, part of the 
system, to bear responsibility for the whole. The present chapter does not get itself 
embroiled in the debate about the concept. Instead, it focuses on instruction that prepares 
pilots to act in ways that improve the joint cognitive system (i.e., the cockpit) to keep 
track of its own activity. It is concerned in particular with incidents that commonly 
are marked by the term situation awareness and with optimizing performances in 
such situations. Researchers can then retain the term but look at the work it does, for 
example, in assisting flight examiners to mark and select episodes from the simulator 
session for the purpose of debriefing them to afford learning and development on the 
part of the pilots. That is, rather than treating situation awareness as a causal construct 
that is used in the attempt to explain an occurrence (e.g., Dekker 2015a), the concept 
is treated here as part of a discourse that indexes and affords certain kind work (e.g., 
on the part of flight examiners). 

It has been noted that pilots generally emerge from the examinations fatigued, often 
failing to remember what they have done during the preceding four-hour examination 
session in the simulator. As the quotation shows, even the most-experienced pilots 
may fail to notice the presence of only some (autofeather) but not other important 
malfunctions, such as the up trim failure of the live engine (î.€., it had failed to increase 
its output and supply the aircraft with maximum power). Such failures to remember 
preceding events or aspects of the events that pilots had never been aware of will 
inherently mitigate what and how they learn from what has happened and what they have 
undergone. It is precisely here that one might expect representational tools—video or 
other multimedia—to offer opportunities that simple verbal debriefing do not provide. 

Of particular interest to the present chapter is situation awareness or, more precisely, 
the instructable nature of situations in which what flight examiners and accident 
investigators mark by stating that there was a “loss of situation awareness”. Because 
situation awareness is thought ofas a skill, a cognitive ability, and sometimes as a mere 
folk psychological concept, the idea may surprise to find that it might be something 
instructable and the result of instructional processes. But this is precisely what is to 
be shown in this chapter. The instructability of situation awareness is particularly of 
interest given that this human factor is cited among the primary causes in most of 
the accidents referred to in this book, including the crashes of the military-operated 
B-737 near Dubrovnik (Croatia), the ATR-72 600 of TransAsia Flight GE235 in Taipei 
(Taiwan), the ATR-72-212A of Lao Air Flight QV 301 near Pakse (Laos), the Fokker 
F100 of Air Bagan Flight W9011 near Heho (Myanmar), or the DHC-8-102 (Dash 8) 
of Ansett Flight 903 near Palmerston North (New Zealand). 

In the preceding quotation, the senior pilot suggests that not only individual pilots 
but in fact the crew as a whole—and, from a joint cognitive system perspective, we 
might say the cockpit—may “not going to be able to pick up all the events”. In the 
case of the Air Bagan flight (Chapter 7), the pilots apparently were not aware of the 
navigation instruments, which would have told them that the aircraft was not on the 
correct outbound and inbound courses and that the aircraft was not stabilized (1.e., it 
was not within the protection area shaded grey in Figure 7.4). The debriefing scenario 
presented and discussed in Chapter 7 is concerned with the joint work of figuring out 
what the pilot was thinking and with understanding the event through the reconstruction 
of an experience (1.e., the pilot-environment unit). In this chapter, the particular issue 
discussed in the debriefing meeting concerns the engine instruments generally and the 
readings thereof during an engine failure specifically. The data used here derive from 
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a recorded debriefing that happens to involve not only a problem similar to one that 
led to the fatal TransAsia GE 235 crash but also the same aircraft type, though in the 
older cockpit variant (ATR 72-500) (Roth and Jornet 2015). 


SITUATION AWARENESS 


In this section, a brief summary of the fundamental positions of the pro and con 
sides of the scholarly discussion about the situation awareness construct is provided. 
Whereas some parts of the community point to the problematic nature of using a 
theoretical construct (1.e., when it is part of mimesis,) to assign causal nature in the 
real, continuously unfolding world witnessed in real time (e.g., Dekker 2015b) others 
conflate the two forms of order by insisting on the concrete reality of situation awareness 
(Endsley 2015), so that the term also names a real-world phenomenon (mimesis, ). 


The Classical Model 


The best-known model of situation awareness contextualizes the phenomenon in a 
more general model of decision-making (Endsley 1994b). In that model, situation 
awareness has perception, comprehension, and projection as its constitutive components 
(see Chapter 1). The output of situation awareness affects decision-making, which in 
turn affects the performance of actions. The outcome of actions is fed back and becomes 
input to situation awareness (see Figure 4.4). On the one hand, the current workload 
affects all three parts of the basically linear process. On the other hand, there are also 
individual characteristics that affect the main components, including preconceptions 
and objectives, individual ability, experience, and training, and doctrine and rules. 
Fundamentally, situation awareness is defined as “the perception of the elements in the 
environment within a volume of time and space, the comprehension of their meaning, 
and the projection of their status in the near future” (Endsley 1994b). It is quite apparent 
from this model that situation awareness is not stable but both affects and is affected 
by (aspects of) the situation (see Figure 1.2). It is in this way that flight examiners in 
the field actually employ the concept and, therefore, clearly distinguish themselves 
from those who view it as a skill that only affects other pilot skills, such as decision- 
making and the technical ability to fly the aircraft so it stays within the limits (Mavin 
and Dall’Alba 2010). Indeed, in the model Endsley presents, individual ability (i.e., 
skills) affects situation awareness. 

Situation awareness clearly is a dynamic construct, which builds on its historical 
trajectory and therefore integrates itself over. That is, it builds and is sustained over 
time; and it also may deteriorate over time when pilots make adverse decisions and 
manage their tasks so that the workload increases to the point of leaving fewer attentional 
resources for maintaining the current level of situation awareness. Thus, for example, the 
pilots in the Ansett Flight 903 case were descending through the published 5% descent 
gradient when the gear down command was called. There was then a problem with the 
undercarriage apparently taking up the attention of both pilots to such an extent that they 
no longer monitored altitude. The exchange with the area control and the problem with 
tracking the checklist from the quick reference handbook increased the workload so that 
the pilots never realized how much altitude they had been losing. A similar incidence 
occurred in 1972 (Eastern Air Lines Flight 401), when all crewmembers of a Lockheed 
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L-1011 focused on a problem with the nose gear indicator and failed to attend to altitude, 
ultimately leading to the loss of 99 lives in the Florida Everglades. In both instances, the 
situation awareness was said to have shrunk spatially and functionally as the demands 
on the pilots’ attention increased. This is why it would be disadvantageous if pilots had 
to think about the go-around in the way of the PF in Chapter 9, where every step of 
the standard operating procedure was an action that is generated by a conscious goal. 
Flying a procedure in this matter constitutes a high demand on attentional resources, 
and thus limits or decreases the capacity to remain aware of other important flight 
parameters. But if the procedure is compiled into a single kinetic melody that unfolds 
once triggered, the attentional resources required by the procedures are minimal freeing 
up attentional resources for other task. 

Perhaps the biggest shortcoming of this model of situation awareness is the 
weight it places on an internal mental model, which is the central component that 
governs everything else. The mental model affects situation awareness by directing 
comprehension, projection, and expectations; it also directs attention to key features 
in the environment. In turn, perception shapes situation awareness in the form of the 
situation model, which in turn directs selection and revision of the mental model. The 
mental model also affects all the other components, including goals, plans, scripts, and 
actions (see Figure 1.2). The view is problematic in the face of the fact that modeling 
in the artificial intelligence literature shows how people, such as short order cooks, 
can competently operate in environments that would exceed the cognitive complexity 
that several agents can handle (Agre and Horswill 1997). Agents do so by letting the 
environment be its own representation, and they find the required information by 
orienting appropriately. This cognitive orientation only requires indices, pointers, or 
what in more technical terms are called deictic codes (Ballard et al. 1997). Breakdowns 
in the situation awareness-related part of decision-making occur at all three levels, 
perception, comprehension, and projection. 

At the lowest level, relevant information available in the environment is not 
perceived or is misperceived (Endsley 1994b). The pilots assessing the performance 
in the Missed Approach scenario (box, Chapter 1) agree that this is not the place where 
the main problem lies, as the PF in the scenario does perceive the weather downwind 
(see Chapter 3). However, the pilots of the Air Bagan flight apparently did not perceive 
the deviation between the automatic direction finder and their heading while on the 
outbound track; similarly, the automatic direction finder would not have been aligned 
with the track on the inbound as it should have been (Figure 7.6). In the TransAsia 
Flight GE235, too, the pilots apparently were not aware of the engine instruments, 
which showed that the live Engine 1 had up-trimmed (TQ = 100%, up from 90%) and 
the propeller on the affected side had autofeathered (i.e., the blade was oriented parallel 
to the flight corresponding to the instrument reading of Np = 0%). 

More frequently, however, comprehension of a perceived environmental fact that 
falls short (Endsley 1994b). In the Missed Approach scenario, some observers conclude 
that the PF may not have comprehended the weather situation although other observers 
suggest at least some level of comprehension is manifested when the PF states that 
the crew had to watch out. Misinterpretation of existing information also falls into this 
category of errors. 

The third level of error occurs when pilots fail to project what they comprehend into 
the future. Thus, for example, the PF in the Missed Approach scenario does not articulate 
the possibility that the aircraft may enter IMC, a situation that would necessitate flying 
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a go-around procedure. If the pilot had projected appropriately, a briefing would have 
followed including a discussion of the direction to be flown. This would be the case 
unless, as seen in the case of the crew in the Ansett Flight 903, the pilots had forgotten 
that they had talked about a relevant issue less than one minute before. At this level, 
errors may occur because something else is required rather than the established habits. 
Research concerning mixed-fleet flying suggests that some of the errors that occur 
when pilots move from one to another cockpit are due to the unfolding of different 
kinetic melodies that jar with the one required in the current cockpit (e.g., Roth et 
al. 2015; Soo et al. 2016). It comes perhaps with little surprise that relevant accident 
reports suggest particular attention to the need for pilots to establish new routines when 
changing from one to another variant of the same aircraft type. Lao Air and TransAsia 
have pilots flying traditional and glass cockpit versions of the same aircraft, and both 
the Lao Air Flight QV301 and the TransAsia Flight GE235 final reports, where situation 
awareness is noted to have been a problem, make reference to the need for more training 
to facilitate the cross over. 


Alternative Approaches 


In aviation practice, situation awareness functions as a master concept, where it serves 
flight examiners to attribute the causal origin of complex events to human operators 
(Chapter 2). Accident investigators, too, employ the construct in virtually the same way. 
Its use orients readers to find fault in the humans rather than in the system as a whole, 
after having ascertained that the manufacturers of the aircraft and its components can 
be abrogated of any fault because the operators have not acted in whatever ways that 
the relevant manuals describe. But some theorists, too, employ the construct in this 
way, unabatedly insisting on its soundness and the pervasive suitability in the field 
(Endsley 2015). In part, confusion arises because the scientific concepts is taken up in 
the everyday discourse of the field (Millot 2015), where scientific research then finds 
itself confronted with its own concepts and, therefore, fails prey to the most serious 
of dangers that scientists can find themselves in: the reification of their own concepts 
(e.g., Bourdieu 1992). A more critical approach is required to escape that circularity. 

In the classical model, situation awareness is the lynch pin in decision-making 
because of its articulatory role between the environment and the mental representation 
thereof. Situation awareness, thus, seems to be a neat concept because it appears to 
explain everything. Indeed, it is too neat, as some intimate: 


The most serious shortcoming of the situation awareness construct as we have 
thought about it to date, however, is that it’s too neat, too holistic and too seductive. 
We heard here that deficient [situation awareness] was a causal factor in many 
airline accidents associated with human error. We must avoid this trap: deficient 
situation awareness doesn’t cause anything. Faulty spatial perception, diverted 
attention, inability to acquire data in the time available, deficient decision-making, 
perhaps, but not a deficient abstraction! (Billings, cited in Dekker 2015a) 


Based on such intimations, some researchers therefore suggest looking for 
explanations that focus on neurology, on the relationship between learning and 
motivation, or on the trade-offs that have to be made in environments where there are 
competing goals (Dekker and Hollnagel 2004). These authors suggest that situation 
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awareness is little more than part of a folk model that is inherently immune to 
falsification. Indeed, if investigators attach the label “situation awareness” to some of 
the actors in an accident ex post facto then they do little more than pseudoscience; such 
cases can easily be challenged in litigation cases based on the lack of credibility alone 
(Dekker 2006). Thus, there is no way research can test whether the pilot statement “where 
are we headed” is in fact due to a loss of situation awareness. As shown and modeled 
in Part A, many individuals watching the Missed Approach scenario do indeed take the 
statement “missed approach was right, isn’t it?” as a manifestation of poor situation 
awareness (see Chapter 3). The classical model of situation awareness is specifically 
targeted in the critique because it cannot be falsified; and if a theoretical model cannot 
be falsified, then it is not very informative. This relationship between objective sense 
and documentary sense has been described as circular, leading some to condemn the 
concept of situation awareness altogether (Dekker 2015b), while failing to realize in 
it a manifestation of the essential nature of human reasoning (e.g., Garfinkel 1967). 

Other scholars clearly orient toward more holistic approaches, looking for problems 
not in the head of individuals or groups, where situation awareness is completely 
lodged somewhere in the brain, but investigate the relationship between pilots and their 
environment (e.g., Sarter and Wood 1994). In this approach, breakdowns are located in 
the human-machine relation. Instead of pursuing the agenda of an overarching situation 
awareness concept, some scholars propose investigating concrete cognitive operations 
in the specific contexts where these occur, including control of attention, mental 
simulation, directed attention, and mental bookkeeping. In an ecological approach, 
situation awareness is not understood in terms of a more-or-less appropriate model in 
the mind of the agent but rather is conceived as interaction; this interaction rather than 
the individual parts constitutes the unit of analysis (Hukki and Norros 1998). Better 
still, situation awareness might be thought of as transactional relation. Decision-making 
reflects the agents’ orientations. These orientations that underlie the decision-making 
frame the situation from the perspective of the agents; they constitute a horizon and 
shape what is perceived and how agents act toward the perceived world. 

The literature on the topic of collective situation awareness, however, lacks 
convergence. There appear to be three main approaches (Millot 2015): (a) collective 
situation awareness is a combination of individual situated awareness and the level of 
mutual awareness that exists between team members; (b) the overlap of the situation 
awareness of the individual team members; and (c) the situation awareness as a 
phenomenon distributed across the system as a whole, including humans and artifacts. 
Whereas the two first approaches are based on individuals as the unit of analysis, only 
the third approach can capture any systemic effect, that is, where there is a new form 
of situation awareness that derives its existence from the system as a whole rather 
than as the result of the sum or the common denominator of all individual situation 
awareness. Errors and breakdown then are emergent properties of the system. As long 
as situation awareness is approached in the first two ways, as something that can be 
found in the heads of individuals, then it can be used as part of the blame game that 
attributes to frontline personnel the brunt of the responsibility (Salmon et al. 2015). 
Thus, the transactional view (see Chapter 1) is a much more elegant and fruitful 
approach to investigating events that are commonly explained by means of individual 
situation awareness. 
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Transasia Airways Flight GE 235 


On February 4, 2015, TransAsia Airways flight GE 235 employing a regional 
aircraft of type ATR72-600 crashed into the Keelung River only 3 nautical miles 
after takeoff, leading to the death of the three pilots present in the cockpit, one cabin 
crew, and 39 passengers. The accident was spectacular, having been videotaped in 
its final phases by the dashboard cameras from several cars that happened to drive 
on Huandong Boulevard (Figure 10.1), which, in this part of Taipei City, follows 
along the riverbanks (Figure 10.2). A preliminary report was published on July 2, 
2015; the final report was made available on July 1, 2016. Multiple organizational 
issues, technical failures, and deficiencies in the training of non-technical skills are 
cited as factors contributing to the accident. The following account is based on the 
initial occurrence investigation of the Taiwanese Aviation Safety Council (ASC 
2015), the final report (ASC 2016), and an account of the final report (Living Safely 
with Human Error 2016). 





Figure 10.1 Instances before crashing into the Keelung River (Taipei), the TR72-600 of TransAsia 
flight GE 235 veered across a highway where it was recorded from several cars. The crash led 
to the death of three pilots, two cabin crew, and 53 passengers. Following an engine fire after 
takeoff (EFATO), the pilot flying had shut off the operating engine (https://www.youtube.com/ 
watch?v=a_DIAPfE8Fg). 


The pilots—licensed to fly both the older ATR 72-500 version with analog 
instruments and the new ATR 72-600 version with electronic flight instrument 
displays (glass cockpit)—continue the flight even though the automatic takeoff power 
control system (ACTPS) is not armed. The standard operating procedures call for 
an abortion of the flight on the 600 (though not on the 500). However, the ACTPS 
arms only a few seconds later and prior to V1; the aircraft thus continues, becomes 
airborne, and the landing gear is retracted. As the augmented map of the fatal flight’s 
trajectory shows (Figure 10.2), seconds after the aircraft turns to the right, the flight 
data recorder (FDR) reports engine 1 (ENG1) to be in up-trim condition and the 
bleed valve is closed. At that time, an ATPCS sequence begins, which includes the 
auto feathering of the affected engine 2 (ENG2). On the engine and warning display, 
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Figure 10.2 Approximate trajectory of TransAsia flight GE 235 with key calls and actions prior 
to the crash. 


the “ENG 2 FLAME OUT AT TAKE OFF” is displayed. The accident investigation 
shows that there had been an uncommanded auto feathering of the ENG2 propeller 
and that all available evidence is consistent with an intermittent discontinuity of the 
auto feather unit and an unstable behavior of the ATPCS. 

Just prior to ENG2 completing feathering, the PF announces, “I will pull 
back engine | throttle” (Figure 10.2), the power lever is pulled back from 75 to 66 
degrees (see Figure 9.2). The PM asks for a second cross check, then states, “Okay, 
engine flameout check” and, after PF replies with “check”, announces “check up 
trim, yes, auto feather yes”, which the PF acknowledges. The PF announces, “pull 
back number one”, at which time the power lever angle is further reduced to 49 
degrees. The PM confirms engine 2 flameout. PF acknowledges, “okay”, but engine 
1 remains at the current angle of 49 degrees. About 5 seconds later, the engine 1 
power lever angle is further retarded to about 34.5 degrees, while the ENG2 power 
lever angle is increased to 86 degrees. A few seconds later, PF announces “number 
one, feather shut off’. The map (Figure 10.2) shows that later in the flight, the PF 
twice commanded “restart engine” and then states the realization, “Wow, pulled 
back wrong side throttle” before restating to “restart engine”. Five seconds later, 
the flight data recorder stopped recording. 

In the case of an engine flameout after takeoff, the TransAsia ATR 72-600 
standard operating procedure memory items begin with the check of the up trim on 
the live engine, auto feather on the failed engine, the bleeds on both engines, and 
landing gear. The emergency procedures for engine flame out at take off begin as 
shown in Figure 10.3. The task sharing makes the PF responsible for the power levers, 
flight path, speed control, configuration, and navigation; the PM is responsible for 
articulating from memory the checklist, executing the required actions, and doing 
anything related to the overhead panel. Thus, the PM is to announce engine flameout, 
PF is to acknowledge it, and then the PM goes through the checks identified as 
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memory items. As noted above, “ENG 2 FLAME OUT AT TAKE OFF” is noted in 
red on the engine and warning display (EWD) display, followed by a shortened form 
of the emergency procedures (Figure 10.3) with main items (left-most aligned items 
in Figure 10.3) in blue and other items in white. The PF was then to make adjustments 
to the attitude of the aircraft such as to accelerate the targeted speed (VFTO). 

With a normally functioning ATPCS, the PF would not have acted until the 
PM announces “Acceleration altitude”, which, in the fatal flight, had been set to 
5,000 ft, whereas the highest elevation reached was only 1,630 ft. 

The emergency procedures show that following the acknowledged call of an 
engine flameout, the PM is to check the up trim of the live engine, the auto feather 
of the failed engine, gear, and bleed valves (some airlines also specify checking 
of the pack valves, which do not appear in the ASC report). A critical part in the 
diagnosis—and the one at the heart of Chapter 10 on the instructable nature of 
situation awareness—concerns obtaining the information from the EWD. This 
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Figure 10.3 The beginning of the emergency operating procedures following an “Engine flame 
out at (after) take off” call. 


is precisely where issues had been observed during this author’s fieldwork but 
concerning the older ATR 72-500 version (Roth and Jornet 2015). The EWD, during 
an engine 2 flameout, would show (according to the simulations conducted by the 
Aviation Safety Council) engine 1 operating at a torque of or above TO = 100%, 
whereas the engine 2 would be operating at a torque of TO = 0% (Figure 10.4). 
At the same time, the propeller rotation speed would be at NP = 100% for engine 
1, whereas the same speed would be drastically reduced (here to NP = 10%) for 
engine 2. In addition, status indicators would show “UP TRIM” above the ENG2 
torque gauge and “AUTO FTR” below the ENG2 digital gauge. (In the ATR 72- 
500 version, as seen in Figure 10.5 below, the propeller rotation speed would also 
be displayed on an analog gauge similar to the torque gauge, and there would be 
no status indicators.) 

The final report concludes: “the accident was the result of many contributing 
factors which culminated in a stall-induced loss of control” (ASC 2016). With respect 
to crew performance, the executive summary states: 
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UP TRIM 





Figure 10.4 The simulations during the investigations showed what the engine and warning display 
(EWD) would have shown on the engine displays: Up trim on Engine 1, no torque on Engine 2, 
and auto feather signal together with a 10% reading on the Np gauge. 


Had the crew prioritized their actions to stabilize the aircraft flight path, correctly 
identify the propulsion system malfunction which was the engine number 2 loss 
of thrust and then take actions in accordance with procedure of engine number 
2 flame out at take off, the occurrence could have been prevented. 


Among the pilot related issues contributing to the fatal crash, the report includes: 


* Following the uncommanded autofeather of engine number 2, the flight crew 
failed to perform the documented failure identification procedure before executing 
any actions. That resulted in pilot flying’s confusion regarding the identification 
and nature of the actual propulsion system malfunction and he reduced power 
on operative engine number 1. 

+ The flight crew’s non-compliance with TransAsia Airways ATR72-600 standard 
operating procedures—Abnormal and Emergency Procedures for an engine flame 
out at take off—resulted in the pilot flying reducing power on and then shutting 
down the wrong engine. 

e Inappropriate flight control inputs by the pilot flying. 

e The loss of power from both engines was not detected and corrected by the crew 
in time to restart the engine. 


In this analysis, the ASC thereby identified what some models of pilot 
performance (such as the one that was used by several airlines in the research reported 
in this book) would be attributed to loss or lack of situation awareness (evidenced by 
confusion, failing to detect power) and a lack of aviation knowledge and procedures 
(non-compliance, taking inappropriate actions). 
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THE COOPERATIVE ORGANIZATION OF SITUATION AWARENESS 
AS AN INSTRUCTIONAL MATTER 


The final incident report concerning the fatal TransAsia Flight GE 235 (February 4, 
2015) was released in 2016. In this accident, the events following an engine flame out 
at takeoff unfolded such that the aircraft eventually crashed into a river killing 43 of the 
57 people on board (see box). As part of the events, the pilot flying shut down the live 
engine, so that the aircraft was left without power. The pilot shut down the live engine 
even though the instruments on the engine and warning display operated correctly, so that 
it should have been easy to distinguish the engine affected from the one still operating. 
The fact that the pilots apparently were unaware of what two sets of instruments 
displayed may appear curious. Of course, it easily lends itself to attribute the ultimate 
crash to human error generally and to situation awareness specifically. Interestingly, 
already prior to the release of the final report, an article focusing on the instructability 
of situation awareness had been published (Roth and Jornet 2015). The issue at heart 
of the article also concerned those very same instruments, the status of which has not 
appeared in the conscious awareness of the pilots, as shown in the quotation of one of 
them at the beginning of this chapter. The case is of particular interest because both 
pilots in the study were very experienced (15 years), the one flying on the right (first 
officer’s seat) also being a captain and flight examiner (nearly 30 years). Incidentally, 
the two TransAsia pilots also were captains, whereas the third pilot flying in the jump 
seat was a first officer on a training flight. In this chapter, the issue surrounding the 
reading of the engine instruments following an engine malfunction after takeoff is 
taken up again, making use of some of the transcriptions that appeared in the original 
article (Roth and Jornet 2015) but in revised form and accompanied by new analyses 
that address the concerns of the present book. 

During the simulator session that preceded the debriefing sequence analyzed below, 
the flight examiner programmed an engine flame out immediately after the aircraft 
had taken off from an airport runway. The flight examiner programmed two additional 
failures, which the pilots would have to find because the standard operating procedure 
requires checking the associated engine instruments (i.e., the torque and Np gauges). 
These engine instruments are situated in the center of the cockpit front panel where they 
are accessible and shared by both pilots (for a view onto the panel see Figure 1.7). The 
torque gauges for the left and right engines are in the top row; the Np gauges, which 
give the propeller rotation in percent, lie below the torque gauges (Figure 10.5). The 
depiction of the instrument also shows that above the pair of torque gauges there is 
a series of lights. Two of these with the inscription “up trim” light up in green color 
when an engine up trims but stay unlit otherwise. Two further lights are dedicated to 
the pitch of the propeller blades. 

The first of the two failures that the flight examiner introduced is an up trim failure, 
which refers to the fact that the system has failed to move the live engine from the 
90% torque mark where it was operating to the 100% required when the other engine 
fails. This automatic up trim occurs because, for takeoff, the pilots arm the automatic 
takeoff power control system (ATPCS), which activates the up trim (and shuts off the 
bleed valve) on the live engine. The flight examiner introduced an autofeather failure as 
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Figure 10.5 This representation features the top part of the engine and warning display that is at the 
center of the episode. The instruments include the torque (top row) and Np (propeller speed) gauges 
for left and right engines. 


the second problem. This is a failure of the ATPCS to move the propeller blades of the 
failing engine parallel to the direction of flight. This blade position reduces air friction, 
prevents further engine damage, and increases the stability of the flight. 

The up trim failure is associated with a failure of the green light to come on and 
an associated torque needle that remains at the 90% mark. An autofeather problem 
reveals itself in a comparison of torque and Np values. If the torque gauge shows 0% 
while the Np gauge shows a value unequal to 0%, an autofeather problem is likely, 
especially when the Np gauge remains at a high value, which indicates that the propeller 
continues to spin. The flight examiner had made a note about discussing this event 
because the two pilots had not addressed the up trim failure in their cockpit talk and 
they had missed to follow the associated procedures. This would have required the pilot 
monitoring to advance both power levers and to check that the live engine indeed was 
operating at the higher torque level. It was one of the facts that fell into the category 
of situation awareness, because the pilots had not been aware of at least two issues (in 
fact, the pilots also had failed to conduct the relevant checks). The flight examiner had 
concluded that there was an issue with situation awareness; the issue could have been 
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avoided had the pilots better managed the procedure by slowing it down, which would 
have helped them notice what they failed to notice.’ 

In the case of an Engine | flame out, after the pilot monitoring announces the event 
and after the PF confirms, the procedure attributes the following actions to the PM: 


e Announce “Engine flame out” 

e Cancel master warning 

e Check live engine UP TRIM on EWD 

+ Announce “ENG 2 up trim” 

e Check failed engine AUTOFTR on EWD 

+ Announce “ENG 1 autofeather” 

e Check ENG 1 & 2 BLEED fault lights illuminated 
e Check PACK VALVE 1 & 2 fault lights illuminated 
+ Announce “Bleed and pack fault lights” 


In any engine flame out event at takeoff, flight examiners orient towards observing 
whether the pilots do what the standard operations describe. In the present instance, 
where the flight examiner also failed the up trim and the autofeather, particular 
attention will be given to the actions that deal with the failed up trim and autofeather. 
As the events described below show, the pilots become aware of the problem with the 
autofeather but not of the second problem (î.€., the failed up trim). The failed up trim 
not being addressed is part of the objective sense that constitutes the evidence for the 
flight examiner’s documentary sense of a problem in the category of situation awareness. 


The Emergence of Flame Out as an Instructable Topic 


During the individual interviews preceding the debriefing meeting, all three participants 
had identified the episode as significant. During the meeting, the discussion surrounding 
the flame out also is marked clearly by an announcement of the engine failure as the 
issue to be talked about, with a particular focus on the first step; the meeting ends with 
a continuation and playing of another segment. The discussion can be roughly divided 
into three parts: talk preceding, during, and following the replay of the episode on the 
debriefing tool. Although the flight examiner had made a mark attributing what has 
happened to situation awareness, the word itself never occurred during the debriefing 
meeting. Instead, the entire sequence concerned with the flame out and the failure to 
fly it according to the standard operating procedure unfolds in terms of practical talk: 
what was done, what is to be done, and what went wrong. It turns out that the debriefing 
tool used in the presentification of the simulator event not only affords pilots a look 
at what has happened but also affords the flight examiner to discover a problem with 
the problem (i.e., an instrument reading that should have been different, a problem 
subsequently attributed to the flight simulator). 

Chapter 5 lays the foundations for understanding debriefing meetings noting that 
(a) they are endogenously organized and performed social events, (b) the transactional 
work making the relation is part of the communication, and (c) agents orient toward 





' Readers may notice that the flight examiner identified management as the root cause, because it pulled 
situation awareness down. Other flight examiner might have suggested that the root cause was lack 
of situation awareness. 
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the production of a social order as a recognizable phenomenon. Of course, this also 
is the case in the present situation: the flame out and the associated phenomena as the 
topic of the debriefing meeting constitute a collective achievement that requires the 
cooperation of all of the participants. The debriefing episode begins when the flight 
examiner announces to be looking at a flame out just after takeoff that had occurred 
for the pilot flying (PF) the aircraft, who also is a flight examiner. The pilot monitoring 
(PM) at the time suggests that the issue was not that clear because the propeller had been 
sitting at 80%, but agrees that it might be beneficial to look at the event. Whereas the 
flight examiner consents, the PM contests, without providing any supportive reasoning, 
and states having “had to think about what engine had failed”. There is therefore 
an issue whether the event should even be discussed. However, the flight examiner 
apparently continues to search for something in the debriefing tool control panel, and 
then announces having found it: “there is the flameout”, adding that the flameout was 
given “with an up trim that had failed and an autofeather fail”. Instead of continuing 
the debate, the examiner invites the pilots to orient to the display monitor. As there are 
no further objections, the flight examiner takes the opportunity articulate a series of 
questions. The nature of the verbal exchange that follows (Fragment 10.1) is typical 
of situations where someone in the know queries one or more persons for knowledge 
assessment purposes. 


Assessment of Extant Knowledge 


Flight examiners who follow an explicit human factors model in pilot performance 
assessment ask themselves how to categorize some problem. In this case, the flight 
examiner’s airline is using the Model for the Assessment of Pilot Performance (Mavin 
and Dall’Alba 2010), in which knowledge of aircraft and procedures is one of the 
“enabling” skills. From this perspective, and assuming that any such knowledge exists 
somewhere in long-term memory, it makes sense to ascertain first what the pilots 
actually know. The typical sequence to ascertain knowledge in school-type and similar 
situations (e.g., museums, outdoor centers) involves a query or initiation, a reply, 
and an evaluation constituting a sequence generally referred to by the acronym IRE 
(e.g., Roth 2015d). This routine has the person in the know in the first- and third-turn 
positions (i.e., I and E), and the person to be tested in the middle turn position (i.e., R). 
The participants therefore have to cooperate for these three parts to align and produce 
the sequence; and it is only when the sequence actually happens that the knowledge 
assessment occurs. What happens in Fragment 10.1 then may be glossed as the flight 
examiner’s examination of the knowledge of the pilots concerning an engine flame out 
after takeoff. This knowledge concerns both standard operating procedure and aircraft. 


Fragment 10.1? 


19 FE : so (0.70) what are the three things you ((PM)) were looking for when 
you have the; (0.25) a flameout; when you=re calling a flameout as a 
monitoring pilot. 


20 (0.28) 


? The transcription conventions used in this chapter are found in Appendix B. 
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21 PM: yea, so you check (0.32) the up trim. 

22 (0.28) 

23 FE : uhhm (0.14) so what are you looking for with up trim. 
24 PM: a hundred percent. 

25 (0.35) 

26 FE : whereabouts? 


27 PM: onthe torque= 


ENG1 ENG2 
=COULDNT W [E CHANGE ONE] THING pat] ma ru 
so top one? | ((3 times EEES 
28 FE: back and forth)) of ae af” TORQUE = 
29 (0.13) 18874 
30 PM: yea. 
31 (0.48) 


32 FE : alright. so not this one here. ((3 x back and forth)) 
33 PM: No. 
34 FE : notthe n=p then ((back and forth)) 





ENG1 


but the torque [((cursor rests)). See 
35 PM: yea —¡EdEJES 
36 PF : should have the up trim lights as well. d „pei 
37 (0.51) 
38 FE : yap. 
39 (2.45) 





The initiation-reply-evaluation routine is apparent throughout the fragment, even 
in places where a person unfamiliar with social analysis may not immediately see it. 
This is apparent in the first three turns, the first of which is offered up as a query: “What 
are the three things . . .?” (turn 19). The middle turn offers up one item: “You check up 
trim” (turn 21). The third term here does not involve an unequivocal “Yes” or “Yea”, 
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but it nevertheless contains an evaluation. First, there is the interjection generally used 
as acknowledgment and affirmation (“Uh hm”), followed by a query that manifestly is 
treated as a way of “digging deeper”, as it concerns a particular thing to be looked for 
(turn 23). In seeking to extend the preceding reply, the query de facto accepts it. As reply 
turns on the part of the pilots are followed by further queries, the entire fragment amounts 
to an investigation of aviation and procedural knowledge. The social “technology” 
for ascertaining the presence of knowledge is that of the triadic IRE sequence, with 
positions taken as described. That the joint work employing this technology has yielded 
its anticipated result will be articulated later in the debriefing meeting when the flight 
examiner formulates that the PM has indeed stated that knowledge before (see below, 
turn 109). If, on the other hand, a flight examiner turn were followed by an assessment, 
then this would amount to change in the hierarchical order through a contestation of 
the operative power-knowledge mechanism (see Chapter 5). 

Another aspect of the event becomes part of the narration. It also concerns a way 
of discovering whether up trim has occurred: a green light just above the torque panel, 
which, in the scenario flown here, would have been on the side of the pilot flying 
seated to the right in the cockpit. Even though the flight examiner has not asked for 
this information, it comes to be articulated in the talk. There is no uptake of the green 
light, so that this statement comes to be treated as not of interest, at least not at this 
point in time. The talk, an important way of accenting the visible, solely is concerned 
with the reading of the torque gauge. As the remainder of the episode shows, the torque 
gauges, as well as the Np gauges, will be the topic of the talk without any mention 
of the green light that would have shown that up trim had occurred. Moreover, the 
standard operating procedure also calls for checks of the pack and bleed lights. These 
aspects of the procedure, too, do not come into play. As a result, only the torque and 
Np gauges and what is to be done when either or both up trim and autofeather do not 
occur following an engine flame out at takeoff. Importantly, because the verbal exchange 
has unfolded in this way, it could indeed serve as the basis for attributing the required 
aircraft and procedural knowledge. This also eliminates the knowledge category as a 
candidate concept in the explanatory discourse; and it makes it more likely that another 
form of explanation, this time related to the awareness of the instrument readings, is 
the description to be entered in any records. Indeed, the flight examiner does make 
repeated reference to having to note some aspect that is revealed as the product of the 
verbal exchange, and that turns out to be different than what had been recorded in the 
flight simulator. 

The debriefing actually does something else. It also establishes where to look for 
and what to see. This work of focusing is in part achieved by means of the cursor that 
moves to places on the depiction of the engine and warning display that are associated 
with particular topics of talk. It is because the cursor moves where something relevant 
can be found that what is found now constitutes the accented visible. Specifically, two 
sets of instruments become salient, and, in one instance, a particular reading required 
for establishing that the associated engine has up trimmed. Talking about the up trim 
also brings about a statement that makes thematic that pilots had not been aware of 
both failures. That is, the PF has already mentioned before that the autofeather had 
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not worked, but now, in what can be heard as a realization and query concerning two 
failures, raises the possibility for both to have occurred. 


Fragment 10.2 

54 (1.88) ((video starts)) (0.91) 
55 PM: did=he=have (0.31) both fail. 
56 (1.92) 








57 FE : well lets just see what (0.45) so=you=talk=me=through=it. 


The flight examiner offers to defer a reply until later, and, in so doing, leaves the 
question about two failures open. At the same time, it treats the preceding turn as a 
question to which it withholds the reply. Here, too, a different mechanism is at play, 
where someone apparently not in the know initiates a query, and another one defers 
a reply thereby also marking to be in the know and in the power that allows replying 
or withholding a reply. But the verbal exchange also has as its outcome that it sets the 
participants up for watching the simulator event with a particular orientation to those 
aspects that had just been talked about. That is, precisely because the torque and Np 
instruments had been the topic of the talk, what they read rather than other aspects 
available on the debriefing tool display are marked as standing out as the accented 
visible over and against anything else that could be talked about. 


The Production of a Factual Account and Awareness 


This subsection describes the work of producing a factual account and, therefore, 
awareness for what has happened. In the process it also constitutes what pilots may not 
have been aware of. To produce such an account, the participants turn to the debriefing 
tool as an aid to make the original event present again. What is shown and perceived on 
the display during the debriefing meeting can then be treated as the objective ground for 
establishing a univocal account of what happened. This work itself involves particular 
forms of joint awareness in the debriefing setting. The specific focus here concerns 
one aspect generally discussed when the term situation awareness is used: the perception 
of events inside the cockpit. This, as seen in the Missed Approach scenario (box, 
Chapter 1), is distinguished from perception and comprehension of events outside the 
cockpit, such as weather, current path of the aircraft, or current position (see Chapter 3). 

To learn from an event, the nature of this event needs to be established. But, as 
seen in Chapter 7, the participants in the debriefing meeting have had partial access to 
what has happened. The flight examiner has set the external conditions, here the engine 
flame out followed by an up trim and an autofeather failure. The flight examiner also has 
perceived the pilots acting and has been aware to a certain extent what the instruments 
have displayed. The pilots, on the other hand, have witnessed the event from within. 
They have been aware of some aspects and may have had some private thoughts that 
are inaccessible to the flight examiner unless articulated in the cockpit talk. To account 
for the event from the perspective of an experience—the unity/identity of person and 
environment—both the personal and environmental aspects have to be articulated and 
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related. The factual account produced in the debriefing meeting constitutes a narrative 
at the level of mimesis,. This factual account must not be confused with the events 
themselves, because it only constitutes the past in the present seen through the lens of 
all intervening lived-experiencing (Figure 6.2). As any representation, it leaves out many 
things, implies a particular perspective, and so on. The narrative at the level of mimesis, 
incorporates as much as is required to initiate teaching | learning for the purpose of the 
professional development of the pilots. Now, a cockpit is a complex environment with 
a large array of instrument panels and actuators in front of the pilots, on the ceiling, 
to the side, and between the two pilots. From the joint cognitive systems perspective, 
all these aspects together form an irreducible unit. Any videotape of the performance 
of the cockpit system would be limited unless the debriefing could be conducted in an 
environment where the entire situation is replayed to the pilots as if they were actually 
sitting in the aircraft. The debriefing tool provides at least some of this information in 
photo-realistic detail, including the engine and warning display. These then can be used 
to establish the course of events as it had unfolded, including exactly those instrument 
readings that the pilots have had at the time. In this fragment, the factual nature of the 
engine failure and the up trim failure are worked out. 

In this session, the flight examiner uses the cursor to direct the pilots to relevant 
instruments while the debriefing tool displays the original event from the flight simulator. 
As in most debriefing sessions, the debriefing tool has been set up to include a window 
for the video recording shot from behind the two pilots, allowing a view of the panel, 
instruments, and actuators situated between them (see Figure 1.6). The pilots are 
afforded seeing and clearly hearing what they have done and said. In addition, there is 
a window showing what the pilots could see outside the aircraft, and a window for the 
instruments (see Figure 8.1). However, the present participants do not attend to any 
of these other windows but only to the engine and warning display generally and the 
torque and Np gauges specifically, particularly the instrument where the cursor pointer 
is placed at any one point in time. The pointer sometimes is further highlighted by the 
fact that a fat white bar extends from its tail horizontally. 

All three participants are oriented toward the monitor when the flight examiner 
starts the replay. Then, precisely when the left torque needle begins to move downward 
from the 90% position, the flight examiner announces “here is the engine failure” and 
points to the left torque gauge, which occurs while the pilot flying in the simulator 
had made the call “gear up” (turn e). The cursor then moves to the torque gauge of 
Engine 2 (right), thereby orienting the participants to reading what it indicates. The 
needle apparently sits at 90%. That is, rather than having moved to 100% where it 
should be if the engine had up trimmed—as the talk has established prior to replaying 
the scene—the needle remains where it has been prior to the engine flame out. (The 
transcription uses lettered turns for everything that has been part of the sound track 
from the video; numbered turns are used for the debriefing talk, which, in addition, is 
marked by a border surrounding the text.) 
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Fragment 10.3 


a PM :  vee=one rotate. 





b (4.50) 
c PM : positive [es | ENGI — ENG2 
d hou::] hou:: ((pitch trim)) 8, EI 
e PF : gear up pb niai 
- - - + wR 
58 FE : [here is the} engine failure Sa Vee Sa 
90.1 A 90.0 a 
f (1.80) 
g bin ((chime)) 
h PM : ihave impaired 1 00.0 a 


and engine ](0.41) uh::::: (0.17) 














59 FE : see that ne Ares eS 
flame=o 4 men 
i bin 
j pe bin | bin bin bin bin bin bin bin ((chime)) 4 00.8 « 
k PF: confirmed = 
ane 
l (0.11) E = 


o” ~120 


100.0 A 


m PM: anduptrim 
n (0.46) 


Less than 2 seconds later, the PM can be heard to call “and up trim”. As shown 
below, the event in the flight simulator continues to unfold. As far as the two pilots 
are concerned, therefore, Engine 2 has up trimmed, as it should have according to 
the specifications. However, the flight examiner orients the pilots to the right torque 
indicator. It still shows 90%. There is therefore a contradiction in the joint cognitive 
system in that one part of it announces the torque to be where it has been, whereas another 
part of the joint cognitive system treats the instrument as showing 100%. At the time, 
this contradiction goes unnoticed. But when the cursor moves to the right instrument, 
accompanied by the “see that” on the part of the flight examiner, an opportunity arises 
for recognizing that up trim has not occurred. That is, there is an opportunity for 
producing an account of the contradiction, and, therefore, a better understanding of 
what has happened and why it has happened as this is reflected through the experience 
of the pilots. In the video, the PM does recognize an autofeather failure a little after 
the end of Fragment 10.3. 
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The participants continue watching the event unfold, when the PF begins to talk 
about the Np, at which point the flight examiner turns the head towards the pilot, and, 
while the former announces the value that the gauge reads, the latter turns the gaze in 
the direction of the monitor (turns 60-61). There is a question concerning which Np 
gauge shows what. It is precisely when the autofeather fails that the Np gauge will 
indicate a value below 8%—as this has happened in the case of TransAsia GE 235, 
where the affected engine did autofeather and the indicator dropped to about 8% (ASC 
2016). The flight examiner points to the Engine 1 Np gauge, which is still sitting around 
80%, and acknowledges this fact (“oh yes” + nod to the PF) with apparent surprise. The 
situation then is formulated as not being right, because, as the videotape has shown in 
the meantime, the pilots had attended to the problem and addressed it. The PM states 
having been very confused, and the flight examiner attributes the issue to the simulator. 
Later in the meeting the flight examiner acknowledges that from the examiner’s seat 
this could not be seen. 




















Fragment 10.4 

i Pe a pas ell ae ee oot eh) pa 
z PM: power lever one forty=five degrees 

61 PM: yea i [ think it was D| 

aa PF: | confirmed | 

ab (0.22) 

62 FE: the n=p is a hundred. 

ac PF: confirmed ENG ENG2 


. EEC1 APCS EEC2 
ad PM: engine number one raver] cn | rar | 
Ea ) (OFF Ea 


[condition lever autofeather fuel shutoff | 














- — a > yd 

63 PM:  [no=no [other side ema still eighty percent| ag TREN | apg Tony” 

eek) | e 

64 PF: my other side 00.8 4 90.0 4 
a T 

aS f 00 E neo 

ae PF: confirmed ES CERS 


78.3 a 100 





af PM: _ [engine] one feathered 
ENG1 ENG2 
65 FE: [oh yes | mor Ares tc 


66 PM: isit a (0) (e) re 
67 FE: i dont think thats right actually 





68 PM: ¡was ve-ry confu:sed by it 
69 FE: i think thats the sim i dont think you would 


see it at eighty percent on a failed engine 
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In this exchange, an account is produced containing both objective facts and the a 
posteriori narration of the PM having been confused by the display. The flight examiner, 
in acknowledging the fact that the Np display could not have been right thereby also 
produces an explanation that supports the confusion in the unfolding narrative. That 
is, in this story line, the PM justifiably may have been confused because the external 
conditions, unbeknownst to the flight examiners, had been different. It is in this joint 
work of accounting that a coherent narrative emerges that would explain to the PM the 
source of the confusion; and the flight examiner learns that the objective facts were 
different from what was assumed, thereby changing the very ground for the assessment 
of the exercise that had been made in the flight simulator and recorded in examiner’s 
notebook. 

In these two fragments, different orientations can be observed. The flight examiner, 
as the cursor in Fragment 10.3 shows, orients toward the Np gauge of the engine that 
has failed to up trim, the issue that the flight examiner has earlier addressed as the 
problem. The two pilots, however, orient toward the Np gauge of the failed engine. 
The pilot statements are in fact instructions for the flight examiner to re-orientate, as 
seen in the fact that the cursor now moves first to the right (Engine 2) then to the left 
(Engine 1) Np gauge. This aspect of the debriefing meeting clearly exhibits an important 
function of the debriefing tool: it assists participants to identify the aspects each party 
is talking about. In the movement of the cursor, first from the torque to the Np gauge 
level and then from the right to the left gauge, the result of the joint orienting work is 
observed. Because the work can be observed, the previously existing misalignment in 
orientation also becomes salient. 

Important in this fragment is that the flight examiner does not dispute or otherwise 
pursue the description of the mental state that the pilot monitoring provides. The process 
of assessment is made difficult because the flight examiner cannot observe everything 
that is happening or is salient to the pilots. Although the videotape replays what pilots 
have been saying and shows in a rather coarse way what can be seen in the dark cockpit 
from behind, many aspects of the flight, such as the state of the instruments, are not 
easily obtained from the videotape, if they can be obtained at all. In making available the 
engine instruments in the same way that these were present to the pilots, the debriefing 
tool literally makes those aspects of the flight present again that either pilots or the 
flight examiner cannot recall or that were never present to them so that they could be 
represented subsequently. By affording to revisit the precise instrument readings during 
the event, the statement about being confused is consistent with instruments that appear 
to be functioning incorrectly. The confusion, arising from an inappropriate instrument 
reading, becomes an intelligible and plausible cause for missing the up trim failure. 
The debriefing tool thereby aides in the constitution of facts independent of what pilots 
or flight examiner remember from or have recorded during the training event. The 
debriefing tool thereby offers resources for constituting common ground and the basis 
for an account that includes facts that all parties can agree on. 


Performing the Loss of Situation Awareness 


The two orders of mimesis, and mimesis, are very different, especially when the second 
is constituted in the form of text. Thus, what really happened and what pilots were 
aware of and what might have been standing out in their individual minds but remained 
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unarticulated when TransAsia GE 235 was between the airport and its final position in 
the river cannot ever be established. Even the cockpit voice and flight data recordings 
cannot take us back: they are representations of things V(A) rather than the real thing 
A at the very moment that they record (as seen in Chapter 6). There is a radical break 
between the irreversibly unfolding original Event, now part of the past not unlike a 
myth, and the modal shift from Event to recording (e.g., Ashmore and Reed 2000). But, 
as shown in Chapter 9, in face-to-face encounters something else may happen. Body 
movements that have occurred during the Event may occur again when a story about the 
mythic Event is produced. Whereas the two movements have different function—e.g., 
turning the head, gazing toward an instrument, and reading what it displays—they 
constitute one movement that reproduces itself. There is therefore a closer tie between 
the Event and the instant in the debriefing room when the narrative account is produced. 
A philosophical analysis of the issue suggests that the inner reproduction of the outer 
perception of an object shows the same phenomenon as the presentification of this object 
(Husserl 1928). Thus, if the pilots made the instrument reading act A internally present 
during the flying episode in the form V(4), then they would be able to access the same 
phenomenon by internally reproducing the reading R(V(A)). But if they never made 
internally present that reading in the original situation, the act itself would be repeated. 
As suggested in Chapter 9, remembering of aspects of the Event may occur when the 
person performs again a movement that has been an integral part. 

Mimesis, interleaves practical understanding that is of the order mimesis, and 
explanation, with an order of the type mimesis, (see Chapter 6). Thus, debriefing also 
is an opportunity for elaborating practical understanding by explicating the reasons 
for unsuccessful performances. The work of repeating a performance occurs when 
the examiner states that what actually happened was that the participants “weren’t 
aware that we didn’t have an up trim”. The flight examiner frames what is to come as 
a reply by pointing to the human factors model used in the airline. Whatever the PM is 
going to answer, thereby, is invited to occur in terms of a statement that includes one 
or more categories that appear in the performance assessment model—.e., situation 
awareness, flying within parameters, decision-making, knowledge, management, 
and communication. In this, the flight examiner initiates an opportunity to develop 
understanding in terms of the theory that the model presents. The work that produces 
a causal narrative of the Event involves joint attention toward the debriefing tool and to 
accent aspects of it to make these visible. The iconic form in which the gauges are made 
present actually affords performing again the different levels of awareness concerning 
the relevant instruments from the engine and warning display. 


Fragment 10.5 


108 PM: probably this ((points to knowledge in the human factors model)) 
1=s=pose. 











109 FE >no: because i=think< | [you knew; 

110 PM: i kne:::w | i know | 

111 FE : you [knew you know 

112 PM Akh | ((laugh)) 

113 PM: — ijust couldnt see it. 

114 FE soa just told me be fore i 
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pressed play that you had to have ((hand ENG1 ENG2 
movement toward monitor)) a hundred Se 
percent up here. pren 


115 PM: KI: (1.23) just couldnt see it (0.40) “really”. apti, 
[(0.35) and i (1.47) had gone past the n=p El o 
and looked at that sitting on eighty (0.51) 008 a 
and i straight away couldnt really figure out 
initially what] ((points to instrument panel)) alo 
() engine for a few minutes had failed. ate 


o” 120 


FAULT | 1 | FAULT 
| | 











78.3 a 





116 FE : yea 


117 PM :  soi=vemore (0.60) (0.22) .hh (0.26) like (0.95) mnea (0.25) so fixated; 
selective (0.32) 


i dont know. (0.65) yea. (0.45) thats | linear 


118 FE well ~ a possibility >and 





i=m not saying that it is< but theres a possibility that you we::re (0.17) 
in a hurry. 


The fragment opens with another instance of the triadic technology for knowledge 
assessment, here not concerning aviation knowledge but knowledge of performance 
assessment and the model that the airline is using. The reply is inappropriate, as per 
the evaluative turn that follows (turn 109). In the same statement, the flight examiner 
also formulates having established before that the pilot has the knowledge. Here, the 
point is not whether the pilot actually has or does not have the knowledge; instead, 
understanding the unfolding of the debriefing meeting as a unique and once-occurrent 
Event requires attentively following what the participants make available to one another. 
It is at this point that the PM articulates not having been able to see “it”, which the next 
turn treats as a reference to the torque gauge reading. It becomes the accented visible 
when the cursor moves to the right torque instrument still showing a 90%. Here, a 
hand movement (turn 114) invites the pilots to orient toward the monitor, which they 
accept by doing so. The flight examiner unmistakably identifies the gauge where the 
100% reading is to be found, which contradicts the 90% actually shown. In that one 
statement and the associated pointer movement and resting place, the earlier unnoticed 
contradiction, which existed in the joint cognitive system at the time of the flight in the 
simulator, now is produced and exists in joint awareness. 

The PM then offers an a posteriori account of what had happened, thereby 
contributing to establishing a narrative that includes first-person as well as third-person 
perspectives. In the present situation, the narrative provides a description that also 
explains why the PM has been aware of the autofeather problem but not the up trim 
failure (i.e., it includes causes and effects). According to this account, the gaze rested 
at the gauge that showed an anomalous and unexpected value. It is this content of this 
perception that the pilot did not understand, preventing diagnosis of which of the two 
engine failed. The account includes a specific value of the Np gauge; and this value still 
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is seen in the appropriate place on the debriefing tool monitor. This failure to understand 
also, according to the account, led to a fixation and selective focus. As pointed out in 
other chapters, there is no way that any such narrative could ever be checked against 
what has happened in the original lived-through Event. The function of the narrative 
is to articulate whatever is deemed to require explanation in an intelligible way. The 
bits and pieces available are organized such that the resulting narrative is intelligible 
and plausible, thereby becoming an acceptable version of the Event. 

While the pilot monitoring presents the first part of the version, the hand is pointing 
in the direction of the engine panels, moving up and down in the way that the required 
gaze would have to do to identify whether autofeathering had occurred (torque = 0%, 
Np = 0%) or had not occurred (torque = 0%, Np > 0%). That is, the PM again refers to 
the Np gauge sitting at 80%, which the three participants had highlighted before and 
described as being higher than it should have been. Whereas other flight examiners do 
in fact point out that during an autofeather failure the gauge sits around 83%, this is 
irrelevant to the unfolding meeting here as per its factual unfolding. The flight examiner, 
however, offers an alternative description: the PM might have been in a hurry. Flight 
examiners frequently talk about pilots reproducing memory items from the standard 
operating procedure without actually doing the required checks. Insiders therefore can 
hear a reference to this culturally specific understanding (i.e., turn 118). 

In this event from the debriefing meeting, situation awareness is not only the 
topic of talk but also performed. The hand movements in the direction of the engine 
instruments invite gaze and orientation to be directed towards the problematic issue, 
articulated verbally as an instrument sitting at 80%. It also is fixed in place, up and 
down where the eyes came to rest during the flight and, therefore, were selective in what 
they were attending to. The hand does not move to the right, consistent with the verbal 
account where the gaze had “gone past the Np” but then could not figure out whether 
this engine had failed. A confusion thereby is performed: If the pilot monitoring had 
problems figuring out which engine had failed, this would have required also looking 
at the torque and Np gauges of the engine on the right. But the pilot states to have been 
fixated on the Np gauge and the up and down movement of the hand describes the gaze 
as being focused on the gauges of one rather than on both engines. 

In this account of the debriefing meeting, it is important to note the characteristic 
orientation of pilots toward praxis, which is distinct from the orientation toward an 
abstract (theoretical) model of pilot performance and the assessment thereof. As 
suggested in Chapter 8, pilots are concerned with flying and how to improve their 
performances rather than with some abstract human factors model that others use to 
explain what they are doing. Thus, the flight examiner invites an explanatory account in 
terms of the theoretical model. The pilot initially offers a tentative reply, the uncertainty 
being made available by means of the hedges (“probably” and “I suppose”) that 
accompany the pointing gesture toward the knowledge category in a diagrammatic 
version of the model. The flight examiner negatively evaluates the reply. In the turns 
that follow, the PM then takes (because given) the opportunity to perform a version 
of what has happened, the uncertainty created by the display still available on the 
iconic representations. That is, the pilot accounts for and performs the event from 
the perspective of a pilot rather than from the perspective of a theorist or theoretical 
practitioner tasked with the assessment in terms of a human factors model. The gap 
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between theory and practice plays itself out right there in debriefing praxis—at least, 
in this situation where airline policy requires an assessment in terms of the explicit 
chosen model and associated measurement instrument. 


Instructing for Attention and Perception 


The principal goal for having debriefing at all is its potential for changing praxis. 
Despite the long history of debriefing and despite its considerable use in high-risk 
situation, reviews and meta-analyses converge on the fact that the effect of video- 
assisted debriefing is marginal to non-existent (e.g., Cheng et al. 2014). Few papers in 
the literature show how to debrief generally and how to debrief for learning specifically 
(Fanning and Gaba 2007). Although the role and function of multimedia appears to 
be poorly understood, the preceding chapters already articulate the many ways in 
which multimedia play an essential role in debriefing and learning. The past findings 
may of course be a result of the lack of studies that investigate debriefing drawing on 
appropriate methods, such as cognitive anthropology and the ethnographic methods that 
are employed in the studies presented here. Simply doing pre- and posttest studies of 
interventions, or experimental studies with posttest only comparisons does not tell us a 
lot about what actually happens in debriefing meetings, how multimedia or technology 
is employed, and what the use of these contributes to enhance learning. 

So far, this chapter is concerned with the joint work of participants to make 
the simulator event present and to elaborate the practical understanding of what has 
happened by developing explanations. The flight examiner has used the term “situation 
awareness” to mark the event; prior to the meeting the examiner had noted that pilots 
would not miss important instrument information if they were implementing appropriate 
management strategies (i.e., slowing down). The purpose for looking at the engine flame 
out at takeoff is to allow these pilots not only to understand what has happened and 
why, but also to learn something from it. That is, these pilots are to walk away from 
the meeting with whatever resources it takes so that if they were to find themselves 
in such a situation again, they would be enabled to act in a way so that any failure to 
up trim is noted. There is therefore a sequence of psychological events that goes from 
representing an event over understanding and explaining it to teaching | learning for 
the betterment of performance when theory (mimesis,) is applied in future practice 
(mimesis,). Whether and when learning occurs cannot be established in the debriefing 
room but only ascertained when the pilots are in the same or similar situation again. 
That is, any instruction occurs at the level of mimesis,, in other words, at the level of 
theory. The debriefing meeting can only lead to a plan for how to address the issue 
should the pertinent situation occur in the future. It cannot address the open relation 
between instruction (plan) and situated acting even if the pilots were to walk away from 
this meeting feeling they have indeed learned something about making sure the up trim 
is checked. As seen in Chapter 9, even though the pilot was aware of how to move 
the power lever and control column at the beginning of a go-around procedure, it took 
several more training sessions in the simulator before the procedure flown conformed 
with the description that the standard operating procedure provides. 

The title of this chapter is “Instructing for Situation Awareness”. The claim here 
however is not that the instructional content is something like situation awareness. The 
term is but a placeholder and marker for the flight examiner. Instead, the term is part of 
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the particular behavior of attending to and marking a specific instrument to be read that 
is taught | learned in this situation. In this fragment, we observe the work of teaching 
| learning concerning the practice of flying the engine flame out at takeoff procedure: 
what the pilot monitoring should do following the failed engine call (pilot monitoring), 
its acknowledgement (pilot flying), and the cancelling of the master warning. The flight 
examiner states that this would have given the pilot flying the opportunity to verify 
the call, finding out that it had been inappropriate. Moreover, putting the finger on the 
instrument actually affords the pilot to gaze and identify the instrument the state of 
which is being called rather than merely calling a state without actually verifying it, 
in the way this was hypothesized to have occurred here. The verbal description comes 
together with hand/arm movements in reference to the instrument presented on the 
debriefing tool. This provides a model for action in this case and, because this also is 
instruction, in all such and similar cases. There are studies suggesting positive effects 
when experts model relevant practices (e.g., LeFlore and Anderson 2009). Although 
such research suggests expert modeling in addition to debriefing, the present case study 
shows that the debriefing tool affords such expert modeling in relevant situations. How 
such modeling occurs and what precisely it does is apparent in Fragment 10.6. 





Fragment 10.6 

120 FE : (1.59) and what (.) and what i (0.41) think [you should do is 
121 PM: u:::m 

122 FE: ac |tuallyslow down 

123 PM: >0:h (just couldn't see that:< u::mm 





124 FE: and actually put your finger (0.15) on what you=re calling. ((points 
towards monitor generically)) so (0.66) if you=re going okay (0.20) 


125 up trim >°well you=re on the other ENG  ENG2 
Est AS ez 
| FAULT FAULT 


side< Cca ca) > 
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Pa a 
y ras, 
is 


a 
vo 
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188.8 a 


E 
apr up 


2 Mp Yo 


126 up trim here (1.02) 


PF would=ave=been perhaps looking 
saying well uh no it isnt. (1.06) and 
and it makes you if you put your 
finger on it just makes you identify it. 
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In the fragment, the examiner describes a way of proceeding that assures that the 
pilot monitoring does not merely go by heart through the list of actions stated in the 
standard operating procedure: monitoring the instruments to check that the automatic 
takeoff power control system (ATPCS) has brought about the required up trim and 
autofeather. The procedure to be used first is given verbally (turn 120-124). Following 
a signal that the instruction is repeated (“So, if you are going . . .”), the flight examiner 
then goes through the sequence performing what is to be done: how to place a finger 
on the instrument prior to making the relevant call (i.e., torque gauge prior to calling up 
trim; and Np gauge prior to calling autofeather). Indeed, the flight examiner begins this 
instructional sequence approaching the engine and warning display instruments from the 
right. But, noting that the PM was sitting on the left, the flight examiner changes to the 
other hand so that the torque gauge is approached from the corresponding side (turn 126). 
The instructional demonstration therefore constitutes a performative modeling of core 
competences that are otherwise spelled out formally in manuals and descriptions. What 
here are deliberate actions simulate the real thing in the presence of iconic presentations 
of the flight instruments and the realistic orientation that the flight examiner takes to 
produce a model specifically for this pilot. The deliberateness of the actions within the 
procedure is modeled by the pauses. The first pause follows the identification of the 
torque gauge by means of pointing and naming the feature from the standard operating 
procedure (i.e., up trim). The second pause follows the description of what the PF could 
have done if the PM had been more deliberate and allowing time for a second look. The 
pauses therefore are iconic with respect to the level of deliberateness, which results in 
a slowing down of the procedural check. 

The fragment is interesting from a joint cognitive system perspective, for it 
articulates a way of improving joint awareness of the system as a whole. This additional 
feature exists in the relation between the pilots, who produce and take up additional 
control instance to cover the possibility that merely following the procedure may, and 
here does, overlook. If the PM moves the hand/finger configuration as shown, the 
movement also is available to the PF, who thereby is afforded the opportunity to do 
a double check (cf. Hutchins and Nomura 2011; Nevile 2004b). That is, although the 
flight examiner here orients toward the PM, as if the situation awareness of this pilot was 
the concern (an in the private mind of the flight examiner this might well have been), 
it provides for a procedure from which both pilots can learn about how to improve the 
perceptual capacities and shared awareness of the events in the cockpit during a critical 
event such as an engine flame out. 

In this situation, the performance of the flight examiner does not just have an iconic 
representational function. That is, the performance does not instruct the correct operation 
by articulating a representation and then transmitting it as informational package (as 
this is theorized in inter actional approaches). If it were a mere representation, then 
there would be a modality shift between what is to be done and how this doing is 
articulated. Such modality shifts exist in the relation between a recipe and the situated 
actions that bring the dish about; and they exist in the relation between the text of 
the standard operating procedure and the actual praxis of pilots on line duty or in the 
simulator. The performance, which occurs against the current display of the debriefing 
tool, provides a presentation of the correct monitoring operation. What can be seen on 
the debriefing tool is similar to the props that appear on the stage of a theatrical play. It 
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is their appearance against this realistic backdrop of the engine instruments that makes 
for the iconic nature of the gestures with respect to the real actions during the flight. The 
orientation toward a realistic modeling is apparent from the first change from a verbal 
description to a gestural description (turn 124 > turn 125) and from the second change 
that corrects the adequacy of the performance with respect to the spatial dimension (turn 
125 > turn 126). Chapter 1 describes the highly spatial nature of the kinetic melodies 
of pilots; and Chapter 3 shows how flight examiners reproduce the spatial specificity 
of the movements that they currently evaluate. In the fragment, an inner contradiction 
comes to exist when the flight examiner approaches the torque gauge from the right 
(turn 125) with what has been observed; and a contradiction exists between what this 
pilot is to learn and what is shown in the gesture from the right. The contradiction 
disappears in the second performance when the hand approaches from the left. Indeed, 
anew contradiction may be thought to exist in the fact that the hand used is not the one 
that the PM needs to use; but this contradiction is not articulated for joint awareness 
and, thus, has no transactional relevance for the way in which this meeting unfolds. 

In this fragment, the flight examiner exhorts the PM to place a finger on the 
instrument currently read (torque), and then, with a temporal delay in the performance, 
models how what can be seen is to be read aloud. Placing the finger, reading “up trim”, 
and leaving a pause with the finger resting on the instrument, improves the awareness 
of the joint cognitive system as a hole. The placement of the finger on the Engine 2 
torque gauge doubles up the identification of the engine concerned, the second existing 
in the required “Engine 2 up trim” (or “Engine 2 up trim failed”) call. That same pause, 
which affords the PF to check the call during the flight, also functions as an aspect of 
the debriefing meeting: it allows the pilots in the meeting to orient to the instrument 
and, in so doing, actually perform now what should have been done then and what is 
to be done in the future. Learning is possible simply because the pilots have performed 
now what they should be performing in the future. 


Summing Up and Articulating the Take-home Message 


The debriefing meeting itself is an event with its own internal order that is produced by, 
and exhibited for the monitoring of, all participants. This order is of the type mimesis, . 
In this meeting, another event becomes the focal entity that also had been of the order 
mimesis,; the result is that there is a lived-experiencing of an experience and thus to 
the genesis of consciousness (Vygotsky 1997a). When the pilots and flight examiner 
arrive in the debriefing room, they already have spent four hours in the simulator flying 
(pilots) or managing the system and evaluating performance (flight examiners). They 
then tend to spend a considerable part of the scheduled one-hour period to talk about a 
series of events—with different distributions as to the amount of talk across pilots and 
flight examiners (see Chapter 5). At the end of the debriefing meeting, the possibility 
therefore arises that so much has happened and been talked about, receding (sinking) 
into the past (see Figure 6.2) and is forgotten because nothing stands out. It therefore 
appears important that some form of summing up has to occur for something to be 
emphasized in awareness so that it becomes a resource to the performance during 
future events. Summing up therefore constitutes a form of presentification of an aspect 
of the present event so that it may internally re-presented in the future. Summing up, 
whether on the part of the flight examiner or the pilots, is sometimes but not always an 
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explicit part of debriefing. This final subsection examines the work by means of which 
the purposes of this episode learned come to be articulated. It is a summary of what 
has been achieved in (a) making present a specific event, (b) analyzing it to understand 
why it has happened, and (c) instructing for the purpose of informing the future. The 
summary also tends to bring about the end of the episode-related discussion, and thereby 
creates the opportunity for moving to another episode to be debriefed. 

Just prior to the fragment reproduced here, the flight examiner makes a statement 
according to which the pilot monitoring ought to verbalize what is happening. The flight 
examiner formulates the possibility that the PM had already noticed that the autofeather 
had failed. There is a hand gesture right after the verb “to notice”, where both hands 
are crossed, each index finger pointing away from the examiner. This can be seen and 
heard as having the order crossed, for the up trim in the exercise had an up trim failure 
on Engine 2, to the right, and an autofeather failure on Engine 1, to the left. Following 
the confirmation on the part of the PM, a statement follows that summarizes what can 
and is to be learned from the event. Indeed, what follows is not merely a statement 
but a performance of what needs to happen, a performance that includes the very hand 
movements that do the job in the flight situation. 


Fragment 10.7 


136 FE : so i think this (.) just slow down and | uptrim (1.09) 
be (.) deliberate about each thing; okay 
up trim (1.09) yes. (0.61) 






y 


autofeather. (1.66) yes.= autofeather (1.66) 


137 PM: =uh hm 
138 (0.77) 
139 FE : um (2.61) >does that fit?< 


N 
140 (0.16) Fai i 
141 PM: yea. (0.21) yea, (1.71) yeai didnt i dont 


think i did really look at it just staring | 
at the n=p. 


In this situation, the instruction models the desired deliberate practice, again 
including the pauses that need to occur for the performance to be recognized to 
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have been deliberate. This deliberation here appears in the form of conversationally 
long pauses—a standard maximum silence being about 1 second (Jefferson 1989)— 
between the naming of the feature attended to and the affirmation of its current status. 
Deliberation also is performed in the form of the associated pointing gesture followed 
by a downward motion, as if the finger was actually tapping the gauge, a motion that 
coincides with the status affirming verbal call. When the event was discussed earlier, 
the flight examiner stated slowing down as a possible course of action that provides a 
solution to the problem that was observed: failing to notice the up trim failure. In this 
fragment, the flight examiner performs slowing down and being deliberate about that 
part of the flight. Addressing the pilots, this performance is designed for them, and as 
performance it is more than words and preaching. As performance, the summary invites 
empathy, that is, it provides an opportunity for the instructed pilots to feel on their part 
what is presented by the actor: “being deliberate about each thing”. 

The instruction for checking the trim status of the live engine is both different 
from and repeats the preceding instruction, that is, the instant when the flight examiner 
actually touches the torque and Np gauges on the monitor of the debriefing tool. A first 
difference exists in the fact that in touching the monitor, the flight examiner moves into 
a virtual space of representation. It is created when a speaker talks and acts oriented 
toward an inscription thereby making it an integral part of the performance (Roth and 
Lawless 2002). Because the inscription here depicts a part of the cockpit, the body 
movement recreates the cockpit in a virtual manner, turning the debriefing room at that 
instant into a theatrical stage. In the space created, with the display on the debriefing tool 
monitor as a prop, a locale is recreated virtually: the cockpit and its directionality come 
to life in the space between the flight examiner and the represented engine instruments. 

When the instruction is repeated in Fragment 10.7, the flight examiner is actually 
turning away from the inscription producing and entering what has been called a 
narrative space. In this space, hand/arm and body movements reproduce movements 
in three-dimensional space as these are found in the situation narratively reproduced 
in the present. In flight examiner performance, the hand/arm movements are the same 
as those found in the cockpit. In this fragment, however, contrary to what is generally 
observed, the flight examiner does not use the hand/arm movements of the pilot whose 
performance is the topic of the talk. On the other hand, if the PM were to see the flight 
examiner as if looking into a mirror, then the movement would be precisely as seen. 
However, the repeat performance precisely mirrors the earlier instruction and, in so 
doing, does not introduce additional complexity and potential for confusion. 

If the flight examiner had merely described in words what the PM should have 
done, the instruction would have resembled any other instruction that one might read 
in a book or hear about in a lecture. The mimetic function of the text exists at the level 
of the plot. Indeed, pilots often make reference to the talking-head nature of debriefing 
sessions and CRM courses. In the present case, the situation is different. The flight 
examiner actually performs what has to be done. The mimetic function here is iconic 
in nature in that the audience sees what is to be done in the same mode as it is to be 
done. In contrast to the sometimes-observed practices where flight examiners emphasize 
assessment and therefore the discourse of the formal model, the flight examiner here 
offers a performance to be directly (iconically) imitated in actual flight situations. In 
performing the procedure, the flight examiner provides a visual model of what the 
words describe. Just as seeing someone put together a piece of assemble-it-yourself 
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furniture appears to make it easier to learn how to do it than following written or pictorial 
guides, the performance provides opportunities for learning because the gap between 
description (theory, plan) and situated acting (praxis) is considerably narrowed. In this, 
there is considerable resemblance between the two flight examiners described, that is, 
the one performs how to move the power levers during a go around (Chapter 9) and 
the present one performing the initiation of the engine flame out at takeoff procedure. 


SITUATION AWARENESS, INSTRUCTION AND SITUATED ACTIONS 
Situation Awareness and Instruction 


Situation awareness is a widely used concept in practice, theory, and research. Although 
unproblematic in aviation practice and accident investigation, the usefulness of the 
concept is subject to considerable debate. Some of the discussion in the literature is 
concerned with the question whether situation awareness actually exists as a natural 
phenomenon to warrant using the corresponding term. Other aspects of the discussion 
concern the usefulness of the term to refer to some asset of individuals (pilots) in the 
light of the fact that the humans are only parts of complex systems that may have a 
property similar to situation awareness at the systems level. In this chapter, a case study 
is presented of how an event might be debriefed so that pilots have an opportunity for 
learning how to better perform a procedure. The case study is particularly relevant given 
the frequent appearance of the situation awareness concept in accident report, where it 
tends to be used to lay (some of) the blame on the pilots. This is particularly so in the 
crash of TransAsia Flight GE 235, where the pilots were confronted with the same kind 
of engine flameout at takeoff, and involving the same aircraft but with a different (glass) 
cockpit (see box). In this accident, the pilot flying realized only seconds before crashing 
that he had shut down the wrong engine: “wow pulled back the wrong side throttle” 
(ASC 2015, original emphasis). The relevance of the present chapter lies in the fact that 
it shows how a flight event marked as loss of (or lacking) situation awareness might 
be addressed in the course of debriefing meetings, and how the behaviors subsumed 
under the concept—taken in the sense of the documentary sense (Chapter 3)—might 
be made explicit in instructional sequences. 

In this chapter, the flight examiner had indexed the event as involving a lowered 
level of situation awareness. In the evaluation, it led to a lowering of the assessment in 
the perception subcategory, with a repercussion on the category as a whole. However, 
and this might be a most important lesson to be taken away from this case study, the 
flight examiner, other than in one instance that was never followed through, refrained 
from talking about and explaining the event in terms of the assessment models—as 
flight examiners in other airlines might do. Instead, all the fragments exemplify the 
orientation of the debriefing meeting toward practical understanding, explaining the 
problem in terms of particular parts of the kinetic melody having been omitted, and then 
articulating an instruction (plan) for how to deal with such situations in the future. The 
instructional strategy focused on management, which slows down the execution of the 
procedure and, in the way instructed, increases the possibilities for trapping omissions. 
For the purpose of debriefing and pilot learning, situation awareness was then only a 
placeholder for tagging the situation and indexing the kinds of actions (operations) to 
be addressed in the virtual performances representing improved praxis. 
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This chapter investigates the structure of debriefing processes generally and the role 
of a debriefing tool specifically. In reviewing, describing, and analyzing the preceding 
simulator events, the past is made present not only in the form of a narrative but in fact 
in the form of performances that also include appropriate backdrop to contextualize and 
therefore ground whatever is said and done. These performances serve as normative 
referents to pilots and flight examiner alike, determining what are to be taken as the 
objective (shared) facts. The performances constitute the shared awareness in the 
debriefing room. As in other cases, it provides pilots with a version of the events that 
they may have forgotten entirely or that they have been unaware of. 

In this episode from the debriefing meeting, work was done to establish that the 
PM knew where to look (procedural knowledge) and what instrument reading should 
be made (declarative knowledge) in the case of the live engine when the other was 
subject to a flame out. In the middle part of the debriefing event, it was unambiguously 
established that the torque did not show the required 100% reading but that the PF had 
indeed confirmed the up trim without being aware that the instrument display did not 
match the description. The flight examiner had taken this as evidence that there was 
a problem with the performance that justified the marker situation awareness rather 
than factual aviation knowledge and standard operation procedures. Establishing this 
distinction is important in the context of this airline, where the flight examiner eventually 
has to judge the performance and justify the assessment in view of the explicit human 
factors model on which the assessment of performance is based. All three participants 
in the debriefing meeting know and have been trained to evaluate pilot performance 
using a specific assessment model; but, as evidenced in the fragments, the mapping 
of facts and categories may not be as self-evident to the PM as for the flight examiner 
(at least for the purposes at hand). Thus, perhaps unsurprisingly for a flight examiner 
attuned to the situation, the explanation and instruction focuses on the actual praxis of 
flying the procedure rather than on the explanatory aspects. 

In the first part of the episode, the joint work of the participants is oriented toward 
establishing whether the under-par performance should be marked under deficient 
knowledge or whether it was due to a failure to perceive what the torque gauge read. 
The only item that the pilot monitoring identifies also has been the one that has had 
a problem; none of the participants had been aware of this fact as a problem. The 
articulation of the problem as such is the result of the exchange; and at this point it is 
an integral aspect of the joint awareness characterizing the meeting. The flight examiner 
then performs and thereby instructs what is to be done to avoid the missing step. The 
performance is such that it makes the missing reading an aspect of the accented visible. 
Moreover, the performance manifests a practical procedure that enhances not merely 
the antecedent performance of the PM—independent of the question whether the 
enhanced state is to be named situation awareness—but especially the performance of 
the joint cognitive system as a whole. If these pilots were acting such that what they are 
doing is adequately described by the flight examiner performance, then evidence exists 
that learning has occurred. That is, in a future simulator session the flight examiner 
assessment in the category of situation awareness would be higher than it has been 
on this day. That assessment then constitutes an index that the system as a whole has 
performed those cognitive functions that are indexed by the term. A shorthand notion 
might then describe that the flight examiner has indeed instructed situation awareness. 
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In the first part of this episode, participants establish a reason why the appropriate 
call was not made. When the flight examiner asks why the pilot monitoring identified to 
have had up trim, the reply suggests categorizing it as a lack of knowledge. However, 
this reply is not accepted (“No”), and a reason for the rejection is provided: the pilot 
actually knows that aspect of the procedure. The flight examiner continues saying that 
the pilot monitoring had articulated this fact during the earlier examination, when stating 
that the torque gauge needed to read 100%. Thus, the problem emerged not because 
of a lack of knowledge. Even before the flight examiner completes the statement, the 
pilot monitoring already suggests not having seen what the instrument read. The PM 
restates definitely not having seen “it”, having gone past the Np gauge, and not having 
been able to identify which engine had failed. 

The flight examiner then suggests a possible chain of events that would explain the 
failure: The PM, being in a hurry, has not read the instrument—leaving unarticulated 
the frequent explanation that pilots merely go through memorized sequences of words 
without actually performing the checks that the words describe. The situated action 
then no longer corresponds to the corresponding verbal description, which also has 
the function of a plan or instruction to do a check of the torque gauge. Indeed, the 
standard operating procedure does distribute checking and announcing across two lines 
(see above): in the first of a pair of lines, the checking is formulated (e.g., “Check live 
engine UP TRIM on EWD”), whereas the second formulates the announcement (i.e., 
“Announce”) and its content (i.e., “Engine 2 up trim”). That is, the pilot not only 
fails doing what the instruction formulates and describes (i.e., “Check”) but also is 
and remains unaware of the contradiction between instruction and situated action. In 
saying “Engine 2 up trim”, the PM actually performs announcing, as instructed by the 
standard operating procedures, but it does not perform the checking up the Engine 2 
up trim. The flight examiner then proposes a course of actions that would slow down 
the perceptual processes and, simultaneously, provide the PF with the opportunity to 
see the instrument being read and notice that the call was wrong. The pause affords the 
possibility to achieve control and feedback into the system (see Figure 1.4), because, 
the instrument being verbalized as instructed, the reading of its state is open to be 
checked by PM and PF alike. While instructing, the flight examiner first points to the 
instrument from afar, but then gets up, moves the hand to the monitor, and places the 
index finger of the right hand on the right (Engine 2) torque gauge of the engine that 
was still running. In this way, the performance brings the required form of action into 
the accented visible, for there now exists an iconic relation between the two movements 
toward the gauge. 

There is a second important aspect to the joint work of bringing about this debriefing 
meeting. The PM, with hindsight and in the context whatever else has happened since, 
tells a story about staring at the engine instruments not being able to identify the failed 
engine because the Np gauge of the failed engine was still sitting around 80%. While 
watching the recorded event, the PF points out that the Np gauge still is showing 80%, 
which it should not. It is at that point that the possible failure of the flight simulator 
is articulated for joint awareness. That is, this is an acknowledgment of the fact that 
it was indeed more difficult to identify which engine had the flameout. Here, after 
having seen the actual instrument readings, the flight examiner acknowledges that this 
had been the situation and that it was hard to assess from his position behind the pilot 
in the simulator what was actually done, looked at, and perceived. But, as the flight 
examiner points out, the pilot monitoring should have verbalized whatever was seen 
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or done, and should have slowed down the procedural sequence. In and through the 
exchange with the pilots, the flight examiner becomes aware of the Np gauge reading. 
This awareness, therefore, is one of the outcomes of their joint debriefing work. As a 
result, the flight examiner revises the assessment of what the pilot has done, which was 
not looking at and misreading the torque gauge but being fixated selectively because of 
the Np gauge problem. It is after reviewing the events that the flight examiner accepts 
this version and puts it on record. 

One aspect of the instructional sequence requires attention. Throughout this book, 
it is noted that expertise in pilot know-how exists in the form of kinetic melodies (flow) 
that unfold without requiring conscious awareness. Some of the debriefing instruction 
intends certain aspects of behavior to unfold on their own without requiring conscious 
awareness, such as the decision-making procedure denoted by means of the acronym 
(F-, T-) DODAR (Chapter 8) or procedures captured in mnemonics such as “Power 
up, Pitch up, Clean up” (Chapters 8 and 9). In the present chapter, the instruction 
appears to work in the opposite direction, as the flight examiner invites the pilot to act 
such that they become consciously aware of some aspects of their environment. This 
exhibits a contradiction: pilots may know their standard operating procedures so well 
that they recite them from memory without flaws while failing to do what the text they 
recite is supposed to describe. In effect, pilots in such situations communicate doing 
something that they do not actually do. In other words, what they say they do is not 
what they actually do. 


Instructing for Future Situated Action 


The episode presented above has an important instructional dimension besides its 
evident assessment-related purpose. The instructed object here is the preparation for the 
future performance of the pilots. The flight examiner instructs the pilots with respect 
to a performance that does not yet exist, its future instantiation, and its appropriate 
deployment in the future. Thus, the plan for the future behavior of the PM was fully 
performed and, thereby, produced and modeled for the purpose to be perceived. The 
performative dimension, with the instruments as a backdrop, introduces a dynamic 
aspect into the instructional sequence; and this dynamic aspect is absent from the words 
in the standard operating procedure that describe appropriate performance following 
the engine flame out at takeoff call. Whereas words in themselves do not have mimetic 
function, the plots they contribute to produce do have such function (Ricceur 1984). 
Bodily performances, however, can have mimetic function of the iconic type. These 
performances differ from merely symbolic movements (e.g., when flight examiners 
use hands to show pitch and attitude of aircrafts) because they are (nearly) identical to 
those required to do the work of flying an aircraft. The performance involves the same 
bodily functions and is constrained by the same spatial and temporal limits as those that 
govern the movements during a flight event. The iconic presentation of the instruments, 
together with the hand/arm gestures, fully make present the process following the engine 
flame out call that should have been performed in the situation and, in respect to future 
instances, is the scenario (plan) to be followed. 
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In this episode, we see the work involved in establishing a double orientation 
inherent in the debriefing practice: past events are analyzed and assessed for the purpose 
of improving future performances. This double orientation is apparent when the flight 
examiner addresses not only the PM to whom the failure and lower situation awareness 
is attributed but also the two pilots generally. Moreover, while pointing to and talking 
about this flameout, the debriefing here constitutes a teaching | learning sequence the 
intended outcome of which is an improvement in the performance of flying this type of 
event from now on (i.e., it constitutes a plan for flying such events). That is, although 
the talk is (initially) about what this pilot did in the simulator, the debriefing meeting 
establishes what any one pilot should do when confronted with some such failure in 
future simulator exercises or on the job. Indeed, the instruction is oriented toward 
improving the performance of all joint cognitive systems in which either pilot will be 
part. This is so to some extent because the instruction is not just articulating what the 
PM is to do but also that the PF is to monitor the relationship between (a) where the 
PM points and what the PM says and (b) the actual state of the relevant instrument or 
signal in the cockpit environment. 

On a first level, pointing here occurs within the activity of debriefing, where it 
is part of the joint work of identifying the nature (torque) and location (top row) of 
an instrument and its reading in the case of the remaining live engine after the other 
had a flame out. In this respect, pointing is an integral part of the work that achieves 
the goal of this particular aspect of the debriefing meeting. Participants have already 
initiated the replay of the event. Therefore, there is also a forward, future orientation. 
All participants are invited to orient toward the instruments, locations, and readings 
once the video starts. Pointing also serves as instruction for where and how to orient 
in the immediate future. The repeated identification of the specific instrument to look 
for and to read diminishes the possibility that there is any remaining ambiguity as to 
where the pilots are to orient. 

There is a second dimension to this instruction: pilots not only are invited to orient 
in the manner established but also to orient in this manner during any future instance 
when such an event occurs during an actual line or simulator flight. This is precisely 
part of the purpose of the biannual recurrent training and examination designed not only 
to assess the pilots but also to prepare them for non-normal events that they might not 
ever actually experience during their career. The two spatial frameworks are different, 
because one is to occur with respect to the instrument displays on the debriefing tool 
as part of the debriefing meeting, whereas the other is in the context of actual flying (in 
future simulator sessions or on the line). But the two frameworks share performative 
aspects. Moreover, in the context of the debriefing meeting, there is a retrospective 
dimension: when the flight simulator event is replayed, it provides an opportunity for 
assessing what the pilots have or have not attended to during the preceding simulator 
session. The exchange between flight examiner and pilots establishes a normative 
framework against which past actions are to be assessed and constitutes an instruction 
for future actions. The pointing gesture therefore is not a simple action but obtains a 
triple function. These kinds of pointing, therefore, are not simple ways of indicating 
something in the surrounding environment but complex semiotic acts in which multiple 
fields come to be juxtaposed simultaneously. 
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A third important function may be observed related to the work of making present 
aspects of a now mythical happening that has slipped and is sinking continuously 
further into the past (Figure 6.2). The pilots are assessed on how they have performed 
what the engine flame out at takeoff procedure states in words. But the process of 
assessment is made difficult because the perceptual capacities of flight examiners are 
inherently limited; and they therefore cannot observe everything that is happening 
or is salient in the environment or to the pilots. Flight examiners always only have a 
fragmented image of the Event, which, therefore, limits what any descriptive account 
can achieve with limitations to the explanations that can be provided. Although the 
videotape replays what pilots have been saying and shows in a rather coarse way what 
can be seen from behind in the darkened cockpit, many aspects of the flight, such as the 
state of the instruments, are not easily obtained, if they can be obtained at all through 
direct observations. Indeed, on the simulator video, flight examiners frequently can be 
seen to lean and even step forward to get a better and less distorted look at the flight 
instruments. In making available the engine instruments in the same way that these 
were present to the pilots, the debriefing tool provides opportunities for making those 
aspects of the flight present again. This allows checking for aspects of the flight that 
either pilots or flight examiners do not or cannot recall or that were never present to 
them in the first place, which makes it impossible for representing them subsequently 
during the debriefing meeting. 


MULTIMEDIA AFFORDANCES IN DEBRIEFING 


Currently available meta-analyses of the role of video and other multi-media tools do 
not have an effect on the (learning) outcomes of debriefing (e.g., Tannenbaum and 
Cerasoli 2013). In this and the two preceding chapters, we observe that the debriefing 
tool does have affordances that play significant cognitive functions in supporting the 
meetings. For example, the debriefing tool affords establishing factual matters that are 
taken up in the assessment of the pilots. Moreover, the debriefing tool affords forms of 
indexical grounding of the verbal accounts produced in the exchanges; and it is through 
these accounts that pilots come to remember or reconstruct events that have forgotten 
or have been unaware of. 


Facts and Fictions 


Adequate assessment requires knowing what has actually happened. However, 
establishing what has happened is difficult for a number of reasons; and is even 
impossible at a certain level. Even flight examiners generally indicate that there are 
many aspects of the situation that they are not and cannot be aware (see Chapter 5). 
Indeed, the flight examiner expresses this to the two pilots in the course of the episode 
featured here, and as a result modifies the way in which one of the pilots was assessed. 
Pilots either forget or are not aware of certain aspects of the flight, making it difficult 
to impossible to establish what has occurred other than what is possible for pragmatic 
purposes (e.g., there is only a limited time available for debriefing). Establishing 
what the facts are and therefore what the basis is for any individual assessment is an 
important aspect of debriefing meetings in which the debriefing tool is available. This 
is particularly so because the events in the flight simulator have their own internal 
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dynamic. This dynamic would be represented in an experience, that is, the unity/identity 
of pilots and environment. But neither pilots nor flight examiners are in the position 
to know exactly what happened, which, in a strong sense, makes impossible a precise 
account of the events. However, in the practice of debriefing, whatever account can 
be produced is taken as sufficient ground for assessment and teaching. This is seen in 
the case of the Heho (Chapter 7) and the present scenarios, both of which are proxies 
for the kinds of things that happen in the cockpit joint cognitive systems involved in 
aviation accidents. The meeting fragments in the early parts of this chapter show the 
joint work of the participants to produce factual evidence to become part of a reasonable 
account. This joint work includes (a) orienting one another to the current instrument 
readings and, in so doing, (b) establishing the events such that everyone present can 
agree on the facts for the purposes of this meeting, the evaluation of the pilots, and the 
instruction that ensued from it. 

Whereas the participants do engage in establishing what actually has happened, their 
inquiries tend to stop whenever they have achieved what can reasonably be achieved 
in the context of a debriefing meeting. Participants do not search any deeper or for 
more detailed understanding in the way that researchers do (including the preceding 
analyses), who dissect recordings, make efforts to reproduce even the most-difficult-to- 
hear passages, and extract from the video the tiniest of movements (e.g., recovering the 
movements on the gauges from a grainy video to ascertain what the participants could 
see but which virtually is invisible in the recording). For example, the participants in 
this episode do not further investigate what the PM is doing in the event while flying 
the engine flame out at takeoff procedure. However, when the video displayed in the 
debriefing tool is magnified several times, one can clearly see that the PM moves 
forward to reach out with the hand until it is near or actually touching the instrument 
panel at the torque and Np gauges (see Scenario 1.1, Figure 1.6). This is evidence that 
the pilot had actually been doing what the flight examiner said needed to be done to 
undercut the possibility of forgetting to read the torque gauge. But, as the preceding 
analyses show, the PM never contests the version implicit in the instruction. Here, 
the flight examiner prefaces the actual instruction saying, “What I think you should do 
is...”, which premises that the PM had not been doing what the subsequent statements 
describe. There are other aspects of the event in the simulator that the participants could 
have been exploring, including the fact that the up trim failure also entailed a failure of 
the up trim signal to light up (green) (for location see Figure 10.5). This failure to light 
up had not been noted. The up trim failure also entailed a failure of the bleed and pack 
fault lights, located on the overhand panel, to light up “Fault” in yellow letters. The 
PM had announced checking the bleed and pack fault lights, and yet these were not lit. 
That is, in the cockpit cognitive system, there was no joint awareness of the absence 
of the green “UP TRIM” and the four yellow “FAULT” lights. In fact, the aircraft 
systems knew but did nothing to make sure that the pilots did so, too. That is, there was 
no process or procedure that trapped the partial nature of awareness. The illumination 
of all these lights would have been triggered only if the up trim had worked. Despite 
of these additional dimensions of interest to a fuller understanding of how the joint 
cognitive system functioned and where it failed, these do not become an issue for the 
practical purposes of this debriefing meeting. 

The tentative nature of the facts is apparent in the frequent uses of hedges and 
markers of uncertainty, including the statements “I think” and “I suppose”. Such hedges 
are modifiers that weaken the degree to which a statement can be heard as an articulating 
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a fact (Latour and Woolgar 1986). These modifiers allow statements to be proven 
incorrect later but without a loss of face to the speaker. Not only pilots but also flight 
examiners use these modifiers. What evolves is a narrative that has some flexibility built 
in, which arises from the uncertainties associated with what otherwise are treated as 
factual statements. On the other hand, in debriefing meetings without the debriefing tool, 
pilots tend to accept the flight examiners’ versions of events. This is so in part because 
the flight examiners also take notes, allowing them to fall back on a recording even if 
it were problematic, whereas the pilots themselves generally admit to have forgotten 
much about what actually has happened and what the instruments have read at the time. 
To another part, contesting the flight examiners’ versions would take more time, which 
is contrary to the stated intentions of many pilots (especially in night sessions) to get 
out of the simulator facility as quickly as possible. When a debriefing tool is available, 
the different versions can be tested. Following this meeting, the PM actually noted as 
stand-out feature that the flight examiner’s version of events did not stand up testing, 
which led to a change in the evaluation of precisely this particular exercise. 


Representation and Indexical Ground for Achieving Joint 
Awareness 


An important part of the debriefing meetings where the debriefing tool available is the 
possibility for making indexical references to the aspects of the cockpit. Relevant to the 
simulator event the participants discussed are two pairs of engine indicators (torque, 
Np). At issue in the meeting is the precise location of the instrument that has to be read 
and that it should indicate 100% rather than the 90% it is seen to show. There is no 
reason to repeat and thereby assert the precise instrument if it were not important and 
at issue. The flight examiner’s query seeks to identify the identity of the torque and 
the instrument pair on the top, while pointing to and moving the cursor back and forth 
the top pair of instruments. As the later parts of the episode show, everything leads up 
to an instructional sequence on how to read this instrument in particular so that the up 
trim failure is trapped in and by the joint cognitive system. 

Analyses of lectures that employ visual representations show that the location 
where the narrative is produced may be thought of as constituted by different spaces 
(Roth and Lawless 2002). When the talk is in reference to the visual representation, 
then it is influenced differently than ifthe speaker is turned away from it. In the former 
instance, the narrative and associated gestures are marked by an orientation to the 
inherently two-dimensional representation, whereas when the speaker is facing the 
audience, a three-dimensional narrative space tends to be at work. In the former instance, 
the representation is the focal object, the figure to stand against the ground, whereas 
in the latter instance, the representation functions as backdrop (ground) against which 
the narration emerges as the figure. Each space comes with its own affordances to the 
deployment of gestures. In the episode, the participants orient each other to the relevant 
space. Thus, for example, the flight examiner moves the hand from an orientation 
toward the pilots until it is oriented toward the monitor and functions as a pointer. In 
one instance, the flight examiner gets up and moves forward toward the monitor until 
the finger actually touches the image of the torque instrument (turns 124-126). The 
configuration therefore recreates the spatial aspects of the event in the way it should 
unfold in the future. 
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In the case of the debriefing meetings without a debriefing tool, the orientation of 
flight examiners and pilots is towards each other—unless some other artifact is involved, 
including cockpit tools (QRH, charts), assessment and action models, chalkboard, or 
cockpit poster (see Chapter 8). Participants are also oriented toward each other when a 
debriefing tool is present but unused. However, there is a dual orientation to the other 
participants and to the debriefing tool in those instances where aspects of the preceding 
simulator sessions are replayed or when the display becomes the topic or referent of the 
talk. There is a participation framework that now includes, and is shaped by, something 
else. Indeed, the debriefing tool functions like witnesses called upon by the court. But 
unlike court witnesses, the debriefing tool remains unchallenged. The analyses show 
that the debriefing tool is indeed used to anchor the debriefing talk: participants oriented 
to or touched a particular pair of instruments in contrast to another pair of instruments. 
One aspect of the debriefing meeting concerns the exact location where the PM should 
have oriented to for the purpose of extracting situation-relevant information. In this, the 
extended sequence establishes a perceptual field of relevance to the exclusion of other 
parts not pertinent to flying the particular type of event. The meeting talk in part serves 
to establish an accented visible, that is, it serves to establish what is to be seen for the 
purposes at hand. The multimedia tool therefore affords joint awareness among the 
debriefing participants to emerge, to be monitored, and to be corrected. This situation 
needs to be considered against the background of other debriefing meetings where this 
kind of interaction is not possible or cannot be taken advantage of. 

To theorize the function of multimedia such as the debriefing tool requires 
considering their position in the different parts of events. Thus, the debriefing tool 
includes a videotape of the pilots at work and records different parameters of the 
flight so that these can be displayed in iconic form. In some sense, the debriefing tool 
establishes an index to the order during the simulator-based events (mimesis, ). It is not 
the now-mythical Event itself but a re-presentation of reality inherently different from 
what had presented itself. The recordings are representations marked by all the flaws and 
shortcomings generally associated with them. But as external representations—called 
inscriptions in many social sciences—they are transportable to new situations. In the 
present situation, the recordings are transported to and accessible in the debriefing room. 
The debriefing also constitutes a reality for the participants, with an associated order that 
they produce, turn to, monitor, and formulate for one another. That is, participants in the 
debriefing meeting produce and exhibit an order of the type mimesis, that differs from 
the order in the cockpit. However, the purpose of the meeting is to produce narrative 
accounts of flight events, which include descriptions of what happened, explanations 
thereof, instruction for flying the event better, and assessments. 

In this joint work associated with these aspects of the debriefing talk, there are at 
least two important functions of the multimedia tool that are apparent and articulated 
across Chapters 8-10. First, there is the witness function. Because the recording is 
like a copy of the original events, it aids in making present again some aspects of 
what happened, while leaving out others (e.g., physical and affective states). Indeed, 
because the recording already re-presences events during the flight that it can function 
as a re-presentation later. Something that has not been present in the awareness of 
or represented by the pilots or flight examiners cannot be present again in the future 
debriefing meeting; that is, to be available in the future, an aspect already has to be 
present in consciousness or be represented during the flight or it will not be available 
later. The recordings therefore are good proxies for what happened, and therefore are 
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treated to be better witnesses than either pilots or flight examiners. The recordings were 
present then and are present now in the same unchanged manner, whereas the human 
participants have already been changed while living through events that have occurred 
since. What can be seen in the recording, therefore, becomes part of the material that 
the debriefing participants use to constitute a mimesis,-type product. But the tool also 
functions as an indexical ground that supports the debriefing participants in the work 
they are doing, that is, producing the meeting and producing an account of the preceding 
flight simulator event. 
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Appendix A 
List of Aviation-related 
Abbreviations 


AFCS Automated flight control system 

AP Autopilot 

ATC Air traffic control 

ATPCS Automatic takeoff power control system 
AUTO FTR Auto feather 

CRM Crew resource management 

DME Distance measuring equipment 

DODAR Diagnose, Option, Decide, Assign tasks, Review 
EGPWS Enhanced ground proximity warning system 
EWD Engine and warning display 

FAA Federal Aviation Administration 

FAF Final approach fix 

F-DODAR Fly, Diagnose, Option, Decide, Assign tasks, Review 
FMS Flight management system 

GA Go-around 

GNSS Global navigation satellite system 

IAF Initial approach fix 

IAS Indicated air speed 

ILS Instrument landing system 

IMC Instrument meteorological conditions 

LOFT Line-oriented flight training 

MDA Minimum descent altitude 

MTOP Maximum takeoff power 

PA Public address (system) 

PF Pilot flying 


PM Pilot monitoring 
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QRH 
R-NAV 
TCAS 
T-DODAR 
VMC 
V/NAV 
VOR/DME 


Quick reference handbook 

Area navigation 

Traffic collision avoidance system 

Time, Diagnose, Option, Decide, Assign tasks, Review 
Visual meteorological conditions 

Vertical navigation 


Very high frequency omnidirectional range with distance measuring 
equipment 


Appendix B 


Transcription Conventions 


Notation 


(0.41) 
() 


((turns)) 


flip it 
you have} to flip 





[so flip it] 


°y’know° 
jUST 


fairly deep 


>i look at the< 
<what happens> 
©) 

(stash?) 

«hh 

hh 


i 


Description 


Time without talk, in seconds 


Period in parentheses marks hearable pause shorter than 0.1 
seconds 


Verbs and descriptions in double parentheses are 
transcriber’s comments 


Square brackets indicate overlapping talk 


Grey highlighted text within square brackets indicates the 
extent of the gesture seen in the offprint presented to the 
right 

Degree signs enclose words spoken with less than normal 
intensity 


Capital letters were spoken with louder than normal 
intensity 

Italicised letters indicate emphases, stress 

Colons indicate lengthening of phoneme, about 1/10 of a 
second per colon 

Angular brackets inward mark faster than normal speech 
Angular brackets outward mark slower than normal speech 
Missing words, one word per question mark 

Word followed by question mark indicates uncertain hearing 
Noticeable in-breath 

Noticeable out-breadth 


Punctuation is used to mark movement of pitch (intonation) 
toward end of utterance, flat, slightly and strongly upward, 
and slightly and strongly downward, respectively 

Equal sign indicates that the phonemes of different words 
are not clearly separated, or latching by a second speaker 


Significant jump of pitch upward, downward 
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